APPLYING THE IRT-M MODEL

The IRT-M model produces a set of latent dimensions whose substantive meaning derives from the theoretical concepts underlying pre-analysis coding rules. Thus, applying the IRT-M model begins with identifying a theory that possesses a set of theoretical concepts that one desires to measure. IRT-M will translate each concept to a corresponding latent dimension and compute a latent position for each data unit on each dimension. To illustrate application of IRT-M, we use a motivating example drawn from Kentmen-Cin and Erisen (Reference Kentmen-Cin and Erisen2017), which offers an overview of anti-immigration attitudes in Europe. Kentmen-Cin and Erisen (Reference Kentmen-Cin and Erisen2017) also calls explicitly, in the abstract, for scholars to “employ methodological techniques that capture the underlying constructs associated with attitude and public opinion,” a task for which IRT-M is ideally suited.

Kentmen-Cin and Erisen (Reference Kentmen-Cin and Erisen2017) elaborate on the manner by which perceptions of threat partially determine one’s attitudes toward both immigration and the EU. We draw three distinct dimensions of threat from their discussion of the literature: a sense of economic threat, a sense of cultural threat, and a sense of religious threat. Together with the two attitudes those threat perceptions are theorized to influence, we thus have five theoretical concepts: economic threat, cultural threat, religious threat, support of immigration, and support of the EU. Each of those five concepts corresponds to a substantively meaningful latent dimension, and the theory indicates that people’s values on the first three latent dimensions influence their values on the last two.

The next step in applying the IRT-M model is to identify a data source comprising items that one believes would be informative about individual data units’ positions along the latent dimensions identified by the theory. In the context of our example, we require data that are informative about each of the three threats and the two attitudes. We identified the Eurobarometer survey for that purpose, and specifically chose the February–March 2021 (94.3) wave due to availability (European Commission 2021). In reading through the survey codebook, it became clear that there was a fourth sense of threat, specific to the COVID-19 pandemic, for which the survey was informative and which might have also influenced attitudes toward immigration and the EU along the same lines as those discussed in Kentmen-Cin and Erisen (Reference Kentmen-Cin and Erisen2017): health threat. Thus, we added that latent dimension to our list, giving us six in all.

After identifying the latent theoretical dimensions of interest and a candidate data source, the next step is to produce the constraint matrix. We accomplish that here by coding the items in the data according to their expected relationships to the underlying theoretical dimensions. Each item in the data is an opportunity for the unit to express its positions along the latent dimensions specified in the theory. In the context of the Eurobarometer survey, an item is a possible answer to a survey question, and a positive response to that item is choosing that answer to the question. Each answer to a question is an item as all items must be dichotomous. Converting all non-dichotomous options to a set of dichotomous items is straightforward. For instance, a five-level feeling thermometer survey question would be broken down into five separate dichotomous items. That is known in the machine-learning literature as One Hot encoding. Similarly, a continuous variable would be made dichotomous through the use of thresholds. In the next section, we discuss how to generalize our model for more general items. In other contexts, responses to items may be votes on bills or resolutions, or the presence of particular elements of treaties, constitutions, or other documents.

After creating an array of dichotomous items, one assigns to each item–latent dimension pair one of four possible values: $ 1,-1 $ , $ 0 $ , or NA. One assigns a $ 1 $ if a positive response to that item would be predicted by a greater value along that latent dimension. Conversely, one assigns a $ -1 $ if a positive response to that item would be predicted by a lesser value along that latent dimension. If one believes that values on that latent dimension do not influence the likelihood of a positive response to that item, one assigns a $ 0 $ . Finally, if one truly has no prior belief about whether and how values on that latent dimension influence the likelihood of a positive response to that item, IRT-M allows the user to assign NA to the item–latent dimension pair.

Returning to our motivating example, consider the latent dimension of economic threat. We would code a survey question that asks about one’s personal job situation as relevant to that underlying dimension. If the survey question had four possible responses, ranging from “very good” to “very bad,” then there would be four item–latent dimension pairs to code. We would code the two “bad” responses as $ 1 $ and the two “good” responses as $ -1 $ , since they would be predicted by more and less perception of threat, respectively. In contrast, a survey question asking for one’s opinion about a common European Asylum system would be coded as $ 0 $ along the economic threat dimension, since we do not view responses to that question as influenced appreciably by one’s sense of economic threat.

In coding the Eurobarometer survey wave, we are implicitly assuming that the theoretical concepts we are trying to measure are constant across both space and time, at least during the time the survey was fielded. We are also assuming that the connection between those concepts and the survey questions is constant across both space and time. We view those assumptions as fair for our motivating example, but what would happen if either were violated?

The assumption of constant theoretical concepts is a necessary one for the application of IRT-M. Should concepts not apply equally across space or should they change over time, then the application of IRT-M should be restricted to those regions or times in which the concepts are uniform. Otherwise, it is not clear what concept one would be measuring. In contrast, it is not a problem for IRT-M if the tie between concepts and items varies across space or time, as long as the coding rules capture that variation. In some cases, there may be appreciable variation in coding rules across time and space (Fariss Reference Fariss2014; Reference Fariss2019). For example, a survey question regarding discrimination might change in meaning across time due to shifts in a country’s demographics, or might mean different things in different countries. In such cases, the coding rules should assign a value to each item–latent dimension pair appropriate to each country, at each time. In other cases, particularly when items are presented to samples of the same population in the same context over time, as they often are for repeated surveys, coding rules may be constant across time. In other words, coding rules should be time and space dependent when necessary.

While we expect exact coding rules to vary between coders—connecting items to theoretical concepts allows for subjective assessment—our simulations, described subsequently, illustrate that IRT-M’s performance is not significantly reduced even when the constraints are only 50%–75% correct. Thus, based on our simulation, the model can still return reasonable estimates even with partially misspecified constraints.

Our set of constraints captures coding for all item–latent dimension pairs in the model, and its use distinguishes IRT-M from unsupervised IRT models. Once coding is complete, one inputs the constraints (in the form of a matrix) and the data into the IRT-M package IRTM. The output of IRT-M is a user-definable number of draws from the posterior distribution of each unit’s position along each latent dimension. In other words, IRT-M maps out the posterior probability distribution capturing each unit’s position in the latent space defined by the theoretical concepts encoded in the constraints. One can average those draws—taking the expected value of the posterior distribution—along each latent dimension for each unit. Doing so produces a distribution of the units’ expected positions in latent space, as in Figures 1 and 2. That captures the distribution of the theoretical concepts in the sample. One can then compare the distributions of different concepts within a sample, or of the same concepts across samples. Individual units’ positions in latent space can also be used in subsequent analyses as individual-level direct measures of theoretical concepts.

Figure 1.

Comparison of Distributions of Latent Dimension Posterior Means

Note: These figures are obtained by applying the traditional kernel density estimator with automatic bandwidth selection to the posterior means obtained by each methods for each the N respondents. Left: IRT-M output, with named dimensions. Right: unconstrained IRT output.

Figure 2.

Comparison of Distributions of Posterior Means of Latent Dimensions by Trust in Media

Note: These figures are obtained by applying the traditional kernel density estimator with automatic bandwidth selection to the posterior means obtained by each methods for each the N respondents. Top: IRT-M output, with named dimensions. Bottom: unconstrained IRT output.

We can use our motivating example to illustrate IRT-M’s output. While there is a great deal one could do with our six latent dimensions—and we provide coding rules, data, and complete model output for those who might want to explore further—we focus on three figures that showcase both what IRT-M can do and some differences between IRT-M and an unconstrained IRT model.

Figure 1 compares the six posterior distributions of units’ positions along the latent dimensions arising from IRT-M (on the top) to those arising from an unconstrained six-dimensional IRT model (on the bottom). It is clear that the distributions of latent positions in the sample seem to vary significantly across theoretical concepts in IRT-M, but do not appreciably vary in the unconstrained IRT model. We will return to that point shortly, but first we highlight the distribution arising from IRT-M most different from the rest, that corresponding to religious threat. Unlike the other five dimensions, the distribution of religious threat is sharply peaked near zero with only a small rise away from that peak. Is it the case that, unlike each of the other perceptions of threat, almost no one in the sample perceives religious threat? To answer that, we can turn to the constraints and the coding rules from which they were derived. What we find is that the February–March 2021 wave of Eurobarometer is a poor data source for measuring religious threat. Only three questions, all regarding whether or not terrorism is the most important issue, are reasonably related to religious threat. Further, during a pandemic, very few people felt that terrorism was the most important issue, leading to little variation in response to those questions. Consequently, we should have little confidence in our measure of the religious threat concept. Instead that latent dimension is capturing a more narrow concept of issue importance. That highlights a point made in the previous section: IRT-M does not find latent dimensions that best predict the data. Rather, it computes positions along a latent dimension using only those items tied closely to the theoretical concept connected to that latent dimension. When there are few or no such items, IRT-M will not produce a good measure of that concept. That is a feature, not a bug, of the approach, and we left that dimension in the analysis to reiterate that important point.

What about the other measures of threat? One way to assess the level of confidence we should have in them is by exploring their construct validity. One way to do that is to disaggregate each distribution so as to enable comparisons to behavioral expectations. Figure 2 displays results for one possible disaggregation, in which the population is split according to trust in different media sources. We would expect, all else equal, that those who do not trust the media would be more likely to perceive threat, particularly cultural threat, whereas those who trust the traditional media would be less likely to perceive threat. That is what we see in the distributions derived from IRT-M, particularly for cultural and health threat, less strongly for economic threat. Further, those who trust all media sources generally perceive threat more similarly to those who trust traditional sources than to those who trust no sources. In contrast, only one of the dimensions derived from the unconstrained IRT model, that for Theta3, approaches that behavior. One can repeat that exercise for different subgroups. For example, we find (not shown) that upper class respondents generally perceive less threat than lower class respondents across the three meaningfully measured threat dimensions. That pattern is not observed in the unconstrained model application. Figure 2 thus reiterates another key point: where there are sufficient informative items in the data, IRT-M produces latent dimensions that possess substantive validity.

Figure 3 extends the point about validity to the attitude measures. For both IRT-M and an unconstrained IRT, it displays a correlation matrix linking the six latent dimensions, as well as four values of trust in the media, a social class variable, and one question directly asking about more border controls. The border control question is coded so that higher values imply less desire for more border controls. We would expect it to be strongly correlated with the Support for Immigration latent variable, as it is both substantively related and 1 of about 10 survey questions that are informative about that latent dimension and coded as such in the constraints. The figure bears out that expectation strongly, while the same is not true for any of the latent dimensions arising from the unconstrained IRT model. Thus, whatever is being captured by those unconstrained dimensions, it is not directly interpretable as support for immigration, one outcome variable of interest.

Figure 3.

Correlations: Latent Dimensions, Media Trust, and Support for Border Controls

Note: Top: IRT-M output, with named dimensions. Bottom: unconstrained IRT output.

Figure 3 also speaks to another characteristic of IRT-M: it models correlated latent dimensions, in order to capture correlated theoretical concepts. IRT models typically assume independent priors over the latent dimensions; yet there are often theoretical reasons to believe that the underlying latent dimensions of interest are correlated. Explicitly modeling the correlation between dimensions, as does IRT-M, avoids a fundamental disconnect between theory, data, and model. That comes through in the figure: the three meaningful threats are all positively correlated with each other, in addition to a lack of trust in the media, while being negatively correlated with trust in traditional media. Those correlations are substantively important, and a model without them would not be able to capture different aspects of threat. In contrast, in the unconstrained IRT, the latent dimensions are not substantially correlated with each other, which forces the concepts they represent to be largely unrelated.

Finally, Figure 3 speaks to the idea in Kentmen-Cin and Erisen (Reference Kentmen-Cin and Erisen2017) that perceived threats are linked to attitudes about immigration and the EU. We find that the three meaningful threats are all negatively correlated with support for both immigration and the EU, though the correlations with the former are weak.

We present the technical specification of IRT-M in the next section, but before doing so, it is worth highlighting one more difference between IRT-M and unconstrained IRT models that is more technical in nature. In an unconstrained IRT model, identification and scaling of the latent dimensions occurs via the use of exogenous information relating to units in the data. For example, ideological ideal points might be taken as given for a small number of legislators. In IRT-M, the constraints both directly allow for model identification and indirectly scale the latent dimensions. The latter occurs automatically in the course of running the model via the creation of anchor points. Anchor points are artificial units that possess extreme positions along the latent dimensions. The constraints characterize behavior consistent with extreme values: if someone has a positive response to every item coded as $ 1 $ and a negative response to every item coded as $ -1 $ for some latent dimension, then it is impossible for a person to show up as more extreme along that latent dimension in the data at hand. Accordingly, the extreme anchor points we construct serve as reference points for the bounds of the space spanned by the latent dimensions. Other, real, units are measured relative to those anchor points, aiding in interpretation of their values along each latent dimension. No exogenous information relating to the units is needed: all required information arises from the pre-analysis step of coding the constraints, and so relates only to the items used by IRT-M to compute latent positions and is theoretically rather than empirically driven. Anchor points are automatically removed prior to output.

METHODOLOGY

We now describe our approach more formally. There are several key points of divergence from existing IRT-based models such as Zeileis, Kleiber, and Jackman (Reference Zeileis, Kleiber and Jackman2008). The first is our pre-analysis coding of the constraints, described earlier. That coding entails conversion of the data to a series of binary (yes/no) items, followed by assignment of a $ 1,-1 $ , $ 0 $ , or NA for each item–latent dimension pair. Each assignment is based on whether or not the theoretically defined latent dimension positively, negatively, or does not predict at all the value of that item. The second point of divergence is that identification in our model is accomplished via our constraints, as we will describe. In contrast, typically multidimensional ideal-point-estimation models require for identification purposes some exogenous information relating to the units in the data, usually the ideal points of several units. The third point of divergence is that we explicitly model the covariance of the latent dimensions in our model. IRT models typically assume independent priors for the latent dimensions and do not model their covariance; yet researchers often have theoretical reasons to believe that the underlying latent dimensions of interest are correlated. By modeling covariance, we allow for closer connections between theory, data, and model.

Formally, begin by considering a set of $ i=1,\dots, N $ units each responding to $ k=1,\dots, K $ binary (e.g., yes/no) items. Let $ {y}_{ik}\in \{0,1\} $ denote unit i’s response to item k, and $ \mathbf{Y}\in {\{0,1\}}^{N\times K} $ be a binary matrix in which row i is unit i’s set of responses to all items, and column k is the vector of all units’ responses to item k, so that each entry corresponds to $ {y}_{ik} $ as just defined. We would like to model units’ responses to items as a function of values along $ j=1,\dots, d $ latent dimensions, denoted for unit i by the vector $ {\boldsymbol{\theta}}_i $ . We will also refer to each $ {\theta}_{ji} $ as a factor j. We choose a two-parameter model (Rasch Reference Rasch1960) for this task, as it is commonly used in applied political science (e.g., Clinton, Jackman, and Rivers Reference Clinton, Jackman and Rivers2004; Tahk Reference Tahk2018):

(1) $$ \begin{array}{rl}\Pr ({y}_{ik}=1)& \hskip-1.3em =g({\boldsymbol{\lambda}}_k^T{\boldsymbol{\theta}}_i-{b}_k),\end{array} $$

where $ \boldsymbol{\lambda} $ is a d-length vector of loadings, $ {b}_k $ is a negative intercept, and g is a link function, mapping from $ \mathbb{R} $ to $ [0,1] $ . The intercept is commonly understood as a difficulty parameter in the psychometric testing literature, and can be seen here as an average rate of “yes” responses to item k. Each entry in the vector $ {\boldsymbol{\lambda}}_k $ is a real value, denoted by $ {\lambda}_{kj} $ . The sign of $ {\lambda}_{kj} $ represents whether a larger factor j (i.e., a larger $ {\theta}_{ji} $ ) will increase or reduce the likelihood of a “yes” response to item k, while the magnitude of $ {\lambda}_{kj} $ represents the overall influence of factor j on the likelihood of responding “yes” to item k. For example, a large positive $ {\lambda}_{kj} $ tells us that units with a large factor j are much more likely to respond “yes” to item k.

The main objective of this article, phrased formally, is to introduce a strategy to encode theoretical information linking latent dimensions and items in the model above, while at the same time resolving its identification issues. We do so by introducing a set of K matrices: $ {\mathbf{M}}_1,\dots, {\mathbf{M}}_K $ . There is one matrix for each item, and each matrix is $ d\times d $ and diagonal. Entries in the diagonal of each matrix are denoted by $ {m}_{kjj} $ and are allowed to either take values in $ \mathrm{\mathbb{R}} $ or to be missing, where missingness is denoted by $ {m}_{kjj}=\mathcal{NA} $ . We then retain the same model in Equation 1, but constrain the loadings $ \boldsymbol{\lambda} $ as indicated in Table 1.

Table 1.

Constraints on Loadings $ \boldsymbol{\lambda} $ from Theory

As Table 1 shows, we constrain our model by (potentially) prespecifying whether the relationship between the response to an item and each latent dimension is positive, negative, zero, or not defined. That accomplishes two fundamental goals. First, it lets one introduce known theoretical connections between items and latent dimensions into the modeling framework. Second, it allows for identification of the model. Before expanding on how our framework accomplishes both of those goals, we give a full description of the model we employ. We choose to adopt a Bayesian specification for our model in Equation 1: we place priors on all of its parameters and express the constraints imposed on $ \boldsymbol{\lambda} $ by $ \mathbf{M} $ as prior items on the former.

The full hierarchical model is as follows:

(2) $$ \begin{array}{ll}{Y}_{ik}\qquad & \hskip-0.6em ={\unicode{x1D540}}_{[{\mu}_{ik}>0]},\end{array} $$

(3) $$ \begin{array}{l}{\mu}_{ik}\sim \mathcal{N}({\boldsymbol{\lambda}}_k^T{\boldsymbol{\theta}}_i\hskip0.35em -\hskip0.35em {b}_k,1),\qquad \end{array} $$

(4) $$ {\displaystyle \begin{array}{l}{\lambda}_{kj}\sim \left\{\begin{array}{ll}{\mathcal{N}}_{\left[0,\infty \right]}\left(0,{m}_{kjj}^2\right),& \hskip0.3em \mathrm{if}\hskip0.3em {m}_{kjj}>0,\\ {}{\mathcal{N}}_{\left[-\infty, 0\right]}\left(0,{m}_{kjj}^2\right),& \hskip0.3em \mathrm{if}\hskip0.3em {m}_{kjj}<0,\\ {}{\delta}_{\left[{\lambda}_{kj}=0\right]},& \hskip0.3em \mathrm{if}\hskip0.3em {m}_{kjj}=0,\\ {}\mathcal{N}\left(0,5\right),& \hskip0.3em \mathrm{if}\hskip0.3em {m}_{kjj}=\mathcal{NA,}\ \end{array}\right.k=1,\dots, K,\end{array}} $$

(5) $$ \begin{array}{rl}{b}_k& \hskip-0.9em \sim \mathcal{N}(0,1),\end{array} $$

(6) $$ \begin{array}{rlll}{\boldsymbol{\theta}}_i& \hskip-1.3em \sim {\mathcal{N}}_d(0,\boldsymbol{\Sigma} ),& & i=1,\dots, N,\end{array} $$

(7) $$ \begin{array}{rlll}\Sigma & \hskip-1.2em \sim {\mathcal{IW}}_d({\nu}_0,{\mathbf{S}}_0),& & \end{array} $$

where $ \unicode{x1D540} $ is the indicator function, $ {\mathcal{N}}_P(\boldsymbol{\mu}, \boldsymbol{\Sigma} ) $ denotes the multivariate normal distribution of size d, with mean vector $ \boldsymbol{\mu} $ and covariance matrix $ \boldsymbol{\Sigma} $ , $ {\mathcal{N}}_{[a,b]}(\mu, {\sigma}^2) $ denotes the univariate normal density truncated between real valued constants a and b, and $ {\mathcal{IW}}_d $ denotes the Inverse-Wishart distribution with $ {\nu}_0 $ degrees of freedom and positive, semi-definite matrix $ {\mathbf{S}}_0 $ . Finally, the vector of length d that is 0 at all positions is denoted by $ 0 $ .

We choose a probit specification for our link function g as this is standard practice in Bayesian IRT modeling (e.g., Zeileis, Kleiber, and Jackman Reference Zeileis, Kleiber and Jackman2008); however, the model above can also be extended to logistic IRT by changing the distribution of $ {\mu}_{ij} $ . We express the constraints imposed on the loading vector, $ \boldsymbol{\lambda} $ , by the M-matrix by varying which prior is chosen for this vector depending on $ \mathbf{M} $ (Equation 4). Both $ {\nu}_0 $ and $ {\mathbf{S}}_0 $ are hyperparameters fixed at standard weakly informative (see, e.g., Gelman et al. Reference Gelman, Jakulin, Pittau and Yu-Sung2008) values in our model. That specification is motivated partly by our application, and partly by the fact that it allows us to solve the identification problems associated with the model in Equation 1 in conjunction with the $ \mathbf{M} $ matrices. Additional discussion of our model specification is available in Appendix A of the Supplementary Material.

A major contribution of our model specification is that it models the covariance between latent factors as a free parameter and the covariance between loadings as the identity matrix. Model identification while allowing correlation between factors requires shifting the assumption of independence to the item loadings in the M matrices. This is, of course, a consequential modeling assumption. However, in the context of many applications of interest to social scientists, underlying factors—such as dimensions of ideology—are often interlinked. In contrast, items—such as survey question answers or elements of texts—are often functionally independent, frequently by design. By modeling independence at the item loading level, we leverage the data generating process’s tendency to produce nearly independent loadings. Our resulting IRT model better fits with many applications of interest. We will expand more on the effect of covariance parameters on model identification and estimation shortly. First, we elaborate on the role of our constraint matrix, M.

Using $ \mathbf{M} $ for Model Identification

Identification issues arise when dealing with models such as that in Equation 1. Here, we show how diagonal matrices such as our constraint matrices can be used to solve those issues, without the need for additional prior distributions or more complex model specifications.

Location and Scale Invariance

Models such as that in Equation 1 are not identified because, for any value of the parameters $ {\mu}_{ik}={\boldsymbol{\lambda}}_k^T{\boldsymbol{\theta}}_i $ and $ {b}_k $ , a different set of parameter values that gives rise to the same likelihood could be constructed with $ {\mu}_{ik}^{\prime }={\boldsymbol{\lambda}}_k^T{\boldsymbol{\theta}}_i+a $ , and $ {b}_k^{\prime }={b}_k-\hskip0.1em a $ , for some real value a. Similarly, for any value of $ {\boldsymbol{\lambda}}_k^T{\boldsymbol{\theta}}_i $ , we could construct a different set of parameters that gives rise to the same likelihood with $ {\boldsymbol{\lambda}}_k^{\prime }={\boldsymbol{\lambda}}_k\cdot a $ and $ {\boldsymbol{\theta}}_i^{\prime }={\boldsymbol{\theta}}_i\cdot \frac{1}{a} $ . Those issues are known as location and scale invariance, respectively, and can be prevented by fixing the mean of $ {\boldsymbol{\theta}}_i $ and $ {b}_k $ , and the variance of $ {\boldsymbol{\lambda}}_k $ . We implement this with our model prior choices in Equations 4–7.

Rotation Invariance

One other identification issue that arises is that, for any $ {\boldsymbol{\lambda}}_k $ and $ {\boldsymbol{\theta}}_i $ , the same exact value of the product $ {\boldsymbol{\lambda}}_k^T{\boldsymbol{\theta}}_i $ can be obtained with the product of different parameter values $ {\boldsymbol{\lambda}}_k^{\prime }={\boldsymbol{\lambda}}_k\mathbf{A} $ and $ {\boldsymbol{\theta}}_i^{\prime }=\mathbf{A}{\boldsymbol{\theta}}_i $ , where $ \mathbf{A} $ is an orthogonal rotation matrix. This implies that $ {\boldsymbol{\lambda}}_k^T{\boldsymbol{\theta}}_i={\boldsymbol{\lambda}}_k^{\prime T}{\boldsymbol{\theta}}_i^{\prime } $ . There are several possible ways to prevent this issue, commonly known as rotation invariance. In general, $ d(d-1) $ constraints must be imposed on the model to prevent this issue (Howe Reference Howe1955). A researcher may want to impose more than $ d(d-1) $ constraints if perceived theoretical clarity allows for this; we discuss this setting in Appendix A of the Supplementary Material. Applications of unconstrained IRT generally achieve identification by specifying, before analysis, values along the latent dimensions for a set of units in the data (see, e.g., Zeileis, Kleiber, and Jackman Reference Zeileis, Kleiber and Jackman2008). Though that approach does encode links between latent dimensions and theoretical concepts, it also requires that the analyst knows the location of some units in the latent space ahead of time, which is not always possible.

Instead, IRT-M imposes the necessary $ d(d-1) $ constraints to prevent rotation invariance via the $ \mathbf{M} $ matrices. There are two ways to do so. First, one can set at least $ d(d-1) $ zeros in the diagonals of all the M-matrices, cumulatively. That is the method we describe above. Formally, this requirement can be stated as

$$ \begin{array}{rll}{\displaystyle \sum_{k=1}^K}{\displaystyle \sum_{j=1}^d}{\unicode{x1D540}}_{[{m}_{kjj}=0]}\ge d(1-d).& & \end{array} $$

Setting $ {m}_{kjj}=0 $ is equivalent to assuming that item k does not load on factor j, or that factor j does not influence the likelihood of responding “yes” to item k. Substantively, that means that one must have a certain amount of prior knowledge regarding the connection between items in the data and the latent dimensions. While one does not need any of the items to load on only one of the latent dimensions, a certain number of items must not load on all the latent dimensions.

Second, one can fix at least $ d(1-d) $ of the $ {\boldsymbol{\theta}}_i $ . Rather than make assumptions about the locations of units in latent space as is commonly done, however, IRT-M instead creates anchor points from the M matrices. Those anchor points rely on assumptions about links between items and latent dimensions, rather than about units, and so their use allows analysts to solve rotation invariance issues without having to make strong assumptions about the units themselves. Anchor points also set the scale for the latent space, as described previously. Relying solely on anchor points for identification requires the $ \mathbf{M} $ matrices to generate two anchors per dimension, according to the procedure described below. However, any combination of $ \mathbf{M} $ matrix constraints and anchor points allows for identification, as long as the total number of constraints is at least $ d(d-1) $ .

The intuition behind anchor points is that a hypothetical data unit that responds “yes” to all items that are positively influenced by one factor j, and “no” to all the items that are not will have a large value of factor j. The formal procedure to create a positive anchor point for one factor j, starting from a set of M-matrices, is as follows:

1. 1. Create a new data point, $ {\mathbf{y}}^{new} $ .

2. 2. For all items $ \ell \in \{{m}_{kjj}>0:k=1,\dots, K\} $ , that is, all items such that the direction of their loading on factor j is known and positive, set $ {y}_{\ell}^{new}=1 $ .

3. 3. For all items $ \ell \in \{{m}_{kjj}<0:k=1,\dots, K\} $ , that is, all items such that the direction of their loading on factor j is known and negative, set $ {y}_{\ell}^{new}=0 $ .

4. 4. For all the remaining items, r, set $ {y}_r^{new} $ to missing.

5. 5. Set $ {\theta}_j^{new}=D $ , where D is some positive constant that is in an extreme of the latent space.

To create a negative anchor for the same factor j, the procedure can be followed exactly, except by setting $ {y}_{\ell}^{new}=0 $ at step 2, $ {y}_{\ell}^{new}=1 $ at step 3, and by making D a negative constant at step 5. In the rare occurrence that an anchor point has exactly the same answers as an actual respondent, then the $ \boldsymbol{\theta} $ for that respondent should also be set to that of the anchor point just created.

Model Extensions

In order to simplify exposition, we have presented our model in the context of a probit-based IRT model; however, our proposed approach can be easily extended to logistic IRT and multinomial IRT and, while we will not discuss it in detail, to standard factor analysis for a real-valued outcome as well. The hierarchical formulation of our model also allows for simple handling of missing outcome data. Finally, our model can be modified so that the covariance matrix of the loadings, rather than the factors, is learned from the data. We discuss that last modification in Appendix A of the Supplementary Material. All of those extended models allow Gibbs sampling formulations for their MCMC posterior sampling procedures, greatly improving convergence speed. Importantly, none of those extensions affect how analysts specify and use M-matrices in our approach, further supporting the flexibility and wide applicability of our proposed method.

Logistic and Multinomial IRT

Our model can be extended to logistic IRT and multinomial IRT by adopting the Polya–Gamma formulation of logistic regression of Polson, Scott, and Windle (Reference Polson, Scott and Windle2013). In a binary logistic IRT, we model our responses as $ \Pr ({Y}_{ik}=1)=\frac{\exp ({\psi}_{ik})}{1+\exp ({\psi}_{ik})} $ , where $ {\psi}_{ik}={\boldsymbol{\lambda}}_k^T{\boldsymbol{\theta}}_i-{b}_k $ . Polson, Scott, and Windle (Reference Polson, Scott and Windle2013) show that the likelihood for one item in this model is proportional to

$$ \begin{array}{rll}{L}_{ik}({\psi}_{ik})\propto \exp ({\gamma}_{ik}{\psi}_{ik}){\displaystyle \int_0^{\infty }}\exp (-{\omega}_{ik}{\psi}_{ik}^2/2)p({\omega}_{ik})d{\omega}_{ik},& & \end{array} $$

where $ {\gamma}_{ik}={Y}_{ik}-1/2 $ and $ {\omega}_{ik} $ follows a Polya–Gamma distribution with parameters $ (1,0) $ . If all the priors on the IRT parameters $ {\boldsymbol{\theta}}_i,{\boldsymbol{\lambda}}_k,{b}_k $ are left as they are in Equations 4–6, then that formulation for the likelihood leads to simple conditionally conjugate updates for all three parameters depending on $ {\omega}_{ik} $ , which also has a conditionally conjugate update. That allows for simple extension of the Gibbs sampler for our existing model to the logistic setting without loss of performance in terms of computation time.

Additionally, that model is easily extended to multinomial IRT settings in which one item can have more options than a yes/no response. Suppose that $ {Y}_{ik} $ can be one of $ \ell =1,\dots, L $ options. In that case, the outcome model becomes: $ \Pr ({Y}_{ik}=\ell )=\frac{\exp ({\psi}_{ik\ell })}{\sum_{r=1}^L\exp ({\psi}_{ikr})}, $ where: $ {\psi}_{ik\ell }={\boldsymbol{\lambda}}_{k\ell}^T{\boldsymbol{\theta}}_i-{b}_{k\ell } $ . As Polson, Scott, and Windle (Reference Polson, Scott and Windle2013) note, that model can also be expressed as a special case of the binary logistic model just discussed, leading to similar conditional updates for all the IRT parameters. Notably, the number of item loadings is now different for each item, as one loading vector $ {\boldsymbol{\lambda}}_{k\ell } $ has to be estimated for each possible response. That implies that it is possible to further constrain such response-level loadings with M-matrices: in that case, the analyst may specify a different $ {M}_{k\ell } $ for each possible answer to a multiple-choice question. The prior on $ {\boldsymbol{\lambda}}_{k\ell } $ is then determined based on such a matrix in the same way that it is in the current model, for binary responses.

Missing Responses

Finally, our current formulation allows for simple inclusion of missing outcomes (responses) by modifying the Gibbs sampling update for $ {\mu}_{ik} $ : if a response is missing, then $ {\mu}_{ik} $ can be drawn from an untruncated normal distribution centered at the conditionally updated mean and variance. That is a common tool in Bayesian inference for probit models, and it is explained in detail in, e.g., Gelman et al. (Reference Gelman, Carlin, Stern and Rubin1995). That approach has the clear advantage of allowing analysts to directly deal with missing data without the need to specify other parameters; however, we note that there is evidence that values imputed this way are biased toward the conditional mean of their imputation distribution (Näf et al. Reference Näf, Spohn, Michel and Meinshausen2023). As we already have incorporated this model extension, we caution analysts who believe that imputation-induced bias might affect their estimates too strongly to make use of other imputation methods before employing IRT-M.

MODEL VALIDATION

In this section, we present an in-depth empirical evaluation of our proposed methodology. We first study the performance of IRT-M on a set of simulated data, where ground truth is known and so can be coded directly into the $ \mathbf{M} $ matrices. Then, we apply our model to roll call data in the U.S. Congress in order to compare our results to those arising from a common IRT application, in a setting in which information on the positions of some legislators in latent ideological space is thought to be known.

Simulated Data

We generate one hundred artificial datasets from the model specified in Equations 2–7. Parameter values for the main simulation parameters are held constant for each simulated dataset at the following values: $ N=500 $ , $ K=50 $ , $ {\mathbf{S}}_0={\mathbf{I}}_{3\times 3} $ , $ {\nu}_0=3 $ . We consider four different numbers of latent dimensions— $ d=2,3,5, $ and $ 8 $ —in order to cover a range of substantively useful values.

Comparison with Other Methods

We compare performance of our model (IRT-M) and a correctly specified set of M-matrices to two other methodologies: principal component analysis (PCA) where the principal component associated with the largest eigenvalue is taken as the latent dimension of interest, and a model similar to the one in Equation 1, but without any M-matrices or any constraints on the respondents, as well as uncorrelated factors and loadings. That second comparison model (IRT) is an implementation often used in applications in political science. We compare IRT-M both with and without correlated factors to those two models. For the IRT-M model with uncorrelated factors, we remove the prior in Equation 7 from the model, and fix $ \Sigma ={\mathbf{I}}_{d\times d} $ .

We first compare the mean squared error (MSE) of estimates obtained across the three methodologies. The MSE captures the average distance between the location of the true value of the theta (position along each latent dimension) for each observation in the data (each data unit) and the model’s estimate of that observation’s theta. As in other contexts, an MSE close to zero indicates that the model tends to estimate thetas close to their true values. The results, averaged across dimensions, are shown in Table 2. We note that assessing performance of latent factor models is a complex problem, and MSE can have problems as a metric (Näf et al. Reference Näf, Spohn, Michel and Meinshausen2023). Due to that, we randomly selected several points from each simulated dataset and inspected them to make sure that average MSE did indeed correspond to estimates closer to their true value: we found that to be the case.

Table 2.

RMSE for $ \boldsymbol{\theta} $ —Averaged over Dimensions, Respondents, and Simulations

Note: Lower is better; best method for each N,K,d in bold. Values are root-mean-square error for estimated versus true latent factors, averaged over d dimensions, N units, and 50 simulations. For Bayesian models, estimates are posterior means computed by averaging over 10,000 posterior samples. All results from Bayesian models are computed from four thousand posterior samples obtained from four parallel MCMC chains after two thousand burn-in iterations.

We see that our proposed methodology performs uniformly better than both PCA and the same model without M-matrices. That is evidence that including M-matrices in the model does indeed lead to superior estimation performance. Allowing for correlated factors also seems to lead to improvements in estimation error, demonstrating the usefulness of that addition to our modeling framework. We report MSE and 95% Credible Interval coverage results for both the latent factors and loadings in Appendix B.

We next compare our model (IRT-M) with the other Bayesian method (IRT), in terms of speed of convergence of the MCMC sampler for estimating the latent factors. To perform that comparison, we use the version of the effective sample size (ESS) statistic proposed in Vehtari et al. (Reference Vehtari, Gelman, Simpson, Carpenter and Bürkner2021). Results are shown in Table 3; larger values denote more information being captured in the Markov chain. We see that IRT-M displays similar or better convergence to simple IRT after the same number of iterations. This is generally true for both the version of our model with correlated factors and the version without; however, the uncorrelated model displays overall best convergence. This is expected, as exploring the posterior of correlated factors is generally harder than exploring that of uncorrelated factors. In addition, Tables 7 and 8 in Appendix B of the Supplementary Material show that our models achieve similar convergence performance according to other widely used MCMC convergence statistics.

Table 3.

ESS Convergence for $ \boldsymbol{\theta} $

Note: Higher is better. Best method for each N,K,d in bold. Values are ESS averaged over N units, d dimensions, and 50 simulations. ESS is a statistic that outputs the number of fully i.i.d. samples that have the same estimation power as the autocorrelated MCMC samples. Here, ESS is computed over 10,000 posterior samples. All results from Bayesian models are computed from four thousand posterior samples obtained from four parallel MCMC chains after two thousand burn-in iterations.

Those results indicate that our proposed approach can achieve performance superior to the standard methodology in terms of MSE without requiring the same strong assumptions and while displaying similar convergence properties. The last is notable because our model requires sampling from a truncated multivariate normal posterior when loadings are direction-constrained by M: that sampling step is nontrivial and requires MCMC tools like rejection or Gibbs sampling in itself (Wilhelm and Manjunath Reference Wilhelm and Manjunath2010). Our convergence comparison shows that this harder sampling step does not affect speed and MCMC mixing performance. Note here that including the prior on variance hurts us in terms of relative convergence, but we still do well despite that.

Performance under Misspecification of M-Matrices

We have seen that our model performs well given a correctly specified set of M-matrices, but in real applications coding of the M-matrices is unlikely to be perfect. Thus, we also assess the robustness of our methodology to misspecification of the M-matrices. We do so in the same simulated-data setting as before, but this time our model is given progressively more misspecified M-matrices. Those misspecified M matrices have two effects, and so misspecification affects the model in two ways. One, it alters elements in the diagonals of the matrices, which alters whether or not a certain item loads on a certain factor. Two, those altered loadings are used to generate anchor points, using the procedure described previously.

Results are shown in Figure 4. The figure shows an expected trend: when the M-matrices are in large part misspecified, model performance is poor, while a correctly specified model displays very low error. The key point, however, is that, in general, even with only half of the correct specification for the M-matrices, our model still performs very well.

Figure 4.

MSE of the Three-Dimensional IRT Model at Progressively More Misspecified M-Matrices

Note: Values on the horizontal axis represent percentages of misspecification of M-matrix diagonals. The leftmost point corresponds to a model in which the M-matrices are completely correct, while the rightmost point corresponds to a model in which the M-matrices are completely misspecified.

Application to Roll Call Data

Our model performs quite well according to reasonable benchmarks, as long as there exists ground truth to capture. Our motivating example shows that IRT-M also produces reasonable results in the absence of ground truth. However, in that example, we had no existing measures with which to compare ours. Therefore, for our final exercise in model validation, we apply the same procedure detailed in our motivating example to congressional roll call data from four sources: the 85th and 109th U.S. House and Senate. Roll call data are commonly used as inputs to ideal-point-estimation techniques (e.g., Aldrich, Montgomery, and Sparks Reference Aldrich, Montgomery and Sparks2014; Clinton, Jackman, and Rivers Reference Clinton, Jackman and Rivers2004; Poole and Rosenthal Reference Poole and Rosenthal1985; Reference Poole and Rosenthal1991; Tahk Reference Tahk2018; Treier Reference Treier2011). Further, those techniques use exogenous information about legislators’ latent ideological positions to identify the models, providing a level of ground truth. In contrast, to apply IRT-M, we will code bills—the items in this context—and make no assumptions regarding legislator positions. We can then compare latent positions derived from applying IRT-M with a two-dimensional latent space to existing ideal point measures such as DW-NOMINATE (DWN) scores, obtainable from voteview.com. That comparison not only reinforces the validity of IRT-M, it also helps us understand better the substantive consequences of employing correlated latent dimensions and varying what gets coded as part of each latent dimension.

The first step in applying IRT-M to roll call data is to specify a set of theoretically informed latent dimensions. Our approach does not fix the number of latent dimensions; however, to compare to DWN, we need to limit the number of latent dimensions to 2. As there is no theoretically unique set of two latent dimensions given the range of substantive topics addressed in Congress, we opted for three different sets of coding rules. Coding rule A uses dimensions corresponding to (1) Economic/Redistribution and (2) Social/Cultural/Civil Rights/Equality. Coding rule B uses dimensions corresponding to (1) Economy/Distribution/Power and (2) Civil Rights. Coding rule C uses dimensions corresponding to (1) Economy/Public Distribution/Power and (2) Civil Rights/Redistribution. Bills not corresponding to any of those topics are coded $ 0 $ for all latent dimensions. The major differences between coding rules are where Redistribution falls and what else gets lumped together with Civil Rights. Appendix C of the Supplementary Material contains a full description of all three coding rules, and we make all coded bills and files to replicate figures available as well.

Figure 5 expresses correlations between different latent dimensions for the 85th and 109th House of Representatives. A similar figure for the Senate is in Appendix D of the Supplementary Material. There are six plots in the figure, one for each House and coding rule combination. Within each plot, the four rows and columns correspond to the first and second latent dimensions derived from DWN and IRT-M. Within each plot are 16 subplots, one for each pair of those four dimensions. The subplots along the diagonal illustrate the distribution of that latent dimension for each of Democrats and Republicans. Above the diagonal, which will be our focus, are correlations between the latent dimension in the column and that in the row, both in the aggregate and broken down by political party. We will focus on those correlations in our brief discussion of model validity, though there is much more one could do with our analysis than we have space for here. High correlations between any two dimensions indicate that those dimensions predict voting behavior similarly.

Figure 5.

Correlations between IRT-M and DW-NOMINATE Ideal Points in the House

Note: Each row/column within each subfigure is one of the latent dimensions estimated either by Nominate or IRT-M. The bottom triangle of each subfigure displays scatterplots with each pair of dimensions on each axis. The diagonal contains density plots for each pair of dimensions. The top triangle contains Spearman correlation coefficients for each pair of dimensions. Solid line, triangle = Democratic. Dashed line, filled circle = Republican.

We first highlight two features of Figure 5 that speak to model validity. One, in all but coding rule B of the 85th House, our two dimensions are highly correlated. In contrast, DWN’s two dimensions are mildly correlated in the 85th House and nearly uncorrelated in the 109th. Given the breadth of bill topics covered in all dimensions we have coded, save for the second dimension in coding rule B which uniquely specifies civil rights, that suggests there may be an underlying factor, such as partisanship, driving voting behavior across a range of bills that may obscure more complex ideological preferences (Aldrich, Montgomery, and Sparks Reference Aldrich, Montgomery and Sparks2014). The increasing correlation of our two dimensions over time is consistent with that point. The difference observed in coding rule B of the 85th House is also consistent with that point, given the known schism in the Democratic party at that time over civil rights.

Two, our first latent dimension is consistently highly correlated with DWN’s first dimension. That suggests that we are capturing a similar theoretical concept to that captured in DWN’s first, economic, dimension. For all but coding rule B of the 85th House, which solely captures Civil Rights, our second dimension is also strongly correlated with DWN’s first, for reasons we have noted. However, for coding rule B of the 85th House, our second dimension is much more highly correlated with DWN’s second dimension. That provides further confidence in the validity of our measures, since that dimension in DWN is typically interpreted as being related to civil rights as well, unlike DWN’s second dimension in the 109th House, which does not share that interpretation.

Together, those two features of Figure 5 support our claim to the validity of our approach, and lend us more confidence that we are able to capture substantive meanings without making assumptions about legislators’ preferences. They also illustrate the substantive effects of two aspects of IRT-M: correlated latent dimensions and coding rules. Modeling correlated latent dimensions lets us capture a scenario in which at least one plausible underlying factor drives more than one latent dimension. The underlying factor does not eliminate the meaning of the two latent dimensions—they still have the substantive meanings our coding rules constrained them to have—but it does suggest a richer causal story, of the type described in Aldrich, Montgomery, and Sparks (Reference Aldrich, Montgomery and Sparks2014). That richer story also suggests that, absent correlated dimensions and our coding rules, one dimension found by an unconstrained IRT model may capture a complex combination of theoretical concepts driven by an underlying factor, while other dimensions capture concepts not present in the theory, and potentially of less importance. It is possible that DWN applied to the 109th House is doing just that. Its first dimension, termed “Economic/Redistributive” on voteview.com, may be capturing the influence of partisanship, while its second dimension, left without a meaning and called “Other Votes” on voteview.com, may be capturing behavior that is not part of the underlying theory of two-dimensional ideological voting.

Using coding rules to capture the link between theory and measure has several implications. One is that, as long as the coding rules are adjusted for variation in meaning across time or space, the substantive meaning of the latent dimensions will remain constant. As an example, in coding rule B, the second dimension captures civil rights in both time periods. We can see that the split in the Democratic party during the 85th House is no longer appreciably present by the time of the 109th House as both dimensions are highly correlated in the 109th House. A second implication is that the items that go into what theoretical concepts each dimension captures fundamentally change what each dimension is measuring. As we saw, isolating civil rights as its own dimension leads to a very different measure than what we obtained under either set of coding rules in which it was combined with other related, but theoretically distinct, concepts.