Rating procedure is crucial in many applied fields (e.g., educational, clinical, emergency). In these contexts, a rater (e.g., teacher, doctor) scores a subject (e.g., student, doctor) on a rating scale. Given raters’ variability, several statistical methods have been proposed for assessing and improving the quality of ratings. The analysis and the estimate of the Intraclass Correlation Coefficient (ICC) are major concerns in such cases. As evidenced by the literature, ICC might differ across different subgroups of raters and might be affected by contextual factors and subject heterogeneity. Model estimation in the presence of heterogeneity has been one of the recent challenges in this research line. Consequently, several methods have been proposed to address this issue under a parametric multilevel modelling framework, in which strong distributional assumptions are made. We propose a more flexible model under the Bayesian nonparametric (BNP) framework, in which most of those assumptions are relaxed. By eliciting hierarchical discrete nonparametric priors, the model accommodates clusters among raters and subjects, naturally accounts for heterogeneity, and improves estimates’ accuracy. We propose a general BNP heteroscedastic framework to analyze continuous and coarse rating data and possible latent differences among subjects and raters. The estimated densities are used to make inferences about the rating process and the quality of the ratings. By exploiting a stick-breaking representation of the discrete nonparametric priors, a general class of ICC indices might be derived for these models. Our method allows us to independently identify latent similarities between subjects and raters and can be applied in precise education to improve personalized teaching programs or interventions. Theoretical results about the ICC are provided together with computational strategies. Simulations and a real-world application are presented, and possible future directions are discussed.

This study incorporates a random forest (RF) approach to probe complex interactions and nonlinearity among predictors into an item response model with the goal of using a hybrid approach to outperform either an RF or explanatory item response model (EIRM) only in explaining item responses. In the specified model, called EIRM-RF, predicted values using RF are added as a predictor in EIRM to model the nonlinear and interaction effects of person- and item-level predictors in person-by-item response data, while accounting for random effects over persons and items. The results of the EIRM-RF are probed with interpretable machine learning (ML) methods, including feature importance measures, partial dependence plots, accumulated local effect plots, and the H-statistic. The EIRM-RF and the interpretable methods are illustrated using an empirical data set to explain differences in reading comprehension in digital versus paper mediums, and the results of EIRM-RF are compared with those of EIRM and RF to show empirical differences in modeling the effects of predictors and random effects among EIRM, RF, and EIRM-RF. In addition, simulation studies are conducted to compare model accuracy among the three models and to evaluate the performance of interpretable ML methods.

Hidden Markov models (HMMs) are popular for modeling complex, longitudinal data. Existing identifiability theory for conventional HMMs assume emission probabilities are constant over time and the Markov chain governing transitions among the hidden states is irreducible, which are assumptions that may not be applicable in all educational and psychological research settings. We generalize existing conditions on homogeneous HMMs by considering heterogeneous HMMs with time-varying emission probabilities and the potential for absorbing states. Researchers are investigating a family of models known as restricted HMMs (RHMMs), which combine HMMs and restricted latent class models (RLCMs) to provide fine-grained classification of educationally and psychologically relevant attribute profiles over time. These RHMMs leverage the benefits of RLCMs and HMMs to understand changes in attribute profiles within longitudinal designs. The identifiability of RHMM parameters is a critical issue for ensuring successful applications and accurate statistical inference regarding factors that impact outcomes in intervention studies. We establish identifiability conditions for RHMMs. The new identifiability conditions for heterogeneous HMMs and RHMMs provide researchers insights for designing interventions. We discuss different types of assessment designs and the implications for practice. We present an application of a heterogeneous HMM to daily measures of positive and negative affect.

This article introduces item response models for rating relational data. The relational data are obtained via ratings of senders and receivers in a directed network. The proposed models allow comparisons of senders and receivers on a one-dimensional latent scale while accounting for unobserved homophilic relationships. We show that the approach effectively captures reciprocity and clustering phenomena in the relational data. We estimate model parameters using a Bayesian specification and utilize Markov Chain Monte Carlo methods to approximate the full conditional posterior distributions. Simulation studies demonstrate that model parameters can be recovered satisfactorily even when the dimensionality of the network is small. We also present an extensive empirical application to illustrate the usefulness of the proposed models for complete and incomplete networks.

Intensive longitudinal data (ILD) collected in mobile health (mHealth) studies contain rich information on the dynamics of multiple outcomes measured frequently over time. Motivated by an mHealth study in which participants self-report the intensity of many emotions multiple times per day, we describe a dynamic factor model that summarizes ILD as a low-dimensional, interpretable latent process. This model consists of (i) a measurement submodel—a factor model—that summarizes the multivariate longitudinal outcome as lower-dimensional latent variables and (ii) a structural submodel—an Ornstein–Uhlenbeck (OU) stochastic process—that captures the dynamics of the multivariate latent process in continuous time. We derive a closed-form likelihood for the marginal distribution of the outcome and the computationally-simpler sparse precision matrix for the OU process. We propose a block coordinate descent algorithm for estimation and use simulation studies to show that it has good statistical properties with ILD. Then, we use our method to analyze data from the mHealth study. We summarize the dynamics of 18 emotions using models with one, two, and three time-varying latent factors, which correspond to different behavioral science theories of emotions. We demonstrate how results can be interpreted to help improve behavioral science theories of momentary emotions, latent psychological states, and their dynamics.

Factor score indeterminacy is a characteristic property of factor analysis (FA) models. This research introduces a novel procedure, regression-based factor score exploration (RFE), which uniquely determines factor scores and simultaneously estimates other parameters of the FA model. RFE uniquely determines factor scores by minimizing a loss function that balances FA and multivariate regression, regulated by a tuning parameter. Theoretical aspects of RFE, including the uniqueness of factor scores, the relationship between observed and latent variables, and rotational indeterminacy, are examined. Additionally, clustering-based factor exploration (CFE) is presented as a variant of RFE, derived by generalizing the penalty term to enable the clustering of factor scores. It is demonstrated that CFE creates cluster structures more accurately than the existing method. A simulation study shows that the proposed procedures accurately recover true parameter matrices even in the presence of error-contaminated data, with lower computational demand compared to existing methods. Real data examples illustrate that the proposed procedures provide interpretable results, demonstrating high relevance to the factor scores obtained by existing methods.

Peer grading is an educational system in which students assess each other’s work. It is commonly applied under Massive Open Online Course (MOOC) and offline classroom settings. With this system, instructors receive a reduced grading workload, and students enhance their understanding of course materials by grading others’ work. Peer grading data have a complex dependence structure, for which all the peer grades may be dependent. This complex dependence structure is due to a network structure of peer grading, where each student can be viewed as a vertex of the network, and each peer grade serves as an edge connecting one student as a grader to another student as an examinee. This article introduces a latent variable model framework for analyzing peer grading data and develops a fully Bayesian procedure for its statistical inference. This framework has several advantages. First, when aggregating multiple peer grades, the average score and other simple summary statistics fail to account for grader effects and, thus, can be biased. The proposed approach produces more accurate model parameter estimates and, therefore, more accurate aggregated grades by modeling the heterogeneous grading behavior with latent variables. Second, the proposed method provides a way to assess each student’s performance as a grader, which may be used to identify a pool of reliable graders or generate feedback to help students improve their grading. Third, our model may further provide insights into the peer grading system by answering questions such as whether a student who performs better in coursework also tends to be a more reliable grader. Finally, thanks to the Bayesian approach, uncertainty quantification is straightforward when inferring the student-specific latent variables as well as the structural parameters of the model. The proposed method is applied to two real-world datasets.

In this article, we propose a series of latent trait models for the responses and the response times on low stakes tests where some test takers respond preliminary without making full effort to solve the items. The models consider individual differences in capability and persistence. Core of the models is a race between the solution process and a process of disengagement that interrupts the solution process. The different processes are modeled with the linear ballistic accumulator model. Within this general framework, we develop different model variants that differ in the number of accumulators and the way the response is generated when the solution process is interrupted. We distinguish no guessing, random guessing and informed guessing where the guessing probability depends on the status of the solution process. We conduct simulation studies on parameter recovery and on trait estimation. The simulation study suggests that parameter values and traits can be recovered well under certain conditions. Finally, we apply the model variants to empirical data.

This article proposes a new statistical model to infer interpretable population-level preferences from ordinal comparison data. Such data is ubiquitous, e.g., ranked choice votes, top-10 movie lists, and pairwise sports outcomes. Traditional statistical inference on ordinal comparison data results in an overall ranking of objects, e.g., from best to worst, with each object having a unique rank. However, the ranks of some objects may not be statistically distinguishable. This could happen due to insufficient data or to the true underlying object qualities being equal. Because uncertainty communication in estimates of overall rankings is notoriously difficult, we take a different approach and allow groups of objects to have equal ranks or be rank-clustered in our model. Existing models related to rank-clustering are limited by their inability to handle a variety of ordinal data types, to quantify uncertainty, or by the need to pre-specify the number and size of potential rank-clusters. We solve these limitations through our proposed Bayesian Rank-Clustered Bradley–Terry–Luce (BTL) model. We accommodate rank-clustering via parameter fusion by imposing a novel spike-and-slab prior on object-specific worth parameters in the BTL family of distributions for ordinal comparisons. We demonstrate rank-clustering on simulated and real datasets in surveys, elections, and sports analytics.

Researchers interested in dyadic processes increasingly collect intensive longitudinal data (ILD), with the longitudinal actor–partner interdependence model (L-APIM) being a popular modeling approach. However, due to non-compliance and the use of conditional questions, ILD are almost always incomplete. These missing data issues become more prominent in dyadic studies, because partners often miss different measurement occasions or disagree about features that trigger conditional questions. Large amounts of missing data challenge the L-APIM’s estimation performance. Specifically, we found that non-convergence occurred when applying the L-APIM to pre-existing dyadic diary data with a lot of missing values. Using a simulation study, we systematically examined the performance of the L-APIM in dyadic ILD with missing values. Consistent with our illustrative data, we found that non-convergence often occurred in conditions with small sample sizes, while the fixed within-person actor and partner effects were well estimated when analyses did converge. Additionally, considering potential convergence failures with the L-APIM, we investigated 31 alternative models and evaluated their performance on simulated and empirical data, showing that multiple alternatives may alleviate the convergence problems. Overall, when the L-APIM fails to converge, we recommend fitting multiple alternative models to check the robustness of the results.