2.1 The Unpooled Model (BAM)

A key question is how to specify priors for the individual-level parameters. One option is to let the parameters be estimated separately, as if they were completely unrelated. This is often referred to as an “unpooled” specification—in contrast to a “pooled” one, which would restrict the parameters to be equal for all respondents (see, e.g., Gelman et al. Reference Gelman, Carlin, Stern, Dunson, Vehtari and Rubin2014). Both Aldrich and McKelvey (Reference Aldrich and McKelvey1977) and Hare et al. (Reference Hare, Armstrong, Bakker, Carroll and Poole2015) used unpooled specifications, and the latter did so by placing wide, uniform priors on the shift and stretch parameters: $\alpha _i, \beta _i \sim \text {Unif}(-100,100)$ . Such priors serve to emulate a maximum-likelihood-based approach, but they also leave the model sensitive to noise, and one of the contributions of this article is therefore to improve this part of the model.

Another key question is how to estimate each respondent’s position, $\chi _i$ , on the same scale as the stimuli, while correcting their self-placement, $V_i$ , for shifting and stretching. For each posterior draw, Hare et al. (Reference Hare, Armstrong, Bakker, Carroll and Poole2015) transform the self-placements as follows:

(2) $$ \begin{align} \chi_i = \frac{V_i - \alpha_i}{\beta_i}. \end{align} $$

This transformation is logically consistent with the likelihood function in Equation (1) and similar to the approach of Aldrich and McKelvey (Reference Aldrich and McKelvey1977). However, the division by $\beta _i$ is problematic because it may lead to division by values that are arbitrarily close to zero, resulting in draws approaching positive and negative infinity. To address this issue, the authors use the posterior median rather than the mean to obtain point estimates of the respondents’ positions. This prevents the most extreme draws from shifting the point estimates around, but it does not resolve the underlying problem. A key contribution of this article is therefore to offer a more satisfactory solution.

A final question is how to identify the parameters. Models of these kinds are typically unidentified, as the latent space can be shifted and stretched while yielding the same likelihood values (see, e.g., Bafumi et al. Reference Bafumi, Gelman, Park and Kaplan2005). To address this issue, Hare et al. (Reference Hare, Armstrong, Bakker, Carroll and Poole2015) fixed two stimulus positions on the latent scale while placing standard normal priors on the others: $\theta _j \sim \text {Normal}(0,1)$ . For the sake of comparison, I retain all these specification choices while translating the model from JAGS (Plummer Reference Plummer2003) to Stan (Carpenter et al. Reference Carpenter2017).Footnote ⁴ I refer to the resulting unpooled Bayesian AM model as the BAM model.Footnote ⁵

2.2 A Limited Hierarchical Model (HBAM₀)

An alternative to the unpooled or completely pooled specifications is a “partially pooled” one, which typically involves a hierarchical model. Such a specification follows naturally if we view each set of individual-level parameters as a sample from a common population distribution. This leads to a hierarchical prior structure where the individual-level parameters are given common prior distributions that themselves have parameters to be estimated. A key point is that a hierarchical model uses information across individuals rather than treating them as unrelated, which typically allows the individual-level parameters to be estimated more accurately given the finite data at hand (see, e.g., Gelman et al. Reference Gelman, Carlin, Stern, Dunson, Vehtari and Rubin2014).

In the present case, replacing the uniform priors on $\alpha _i$ and $\beta _i$ also offers an alternative way to identify the model. We can select one out of the many possible transformations of the latent space by setting specific means or medians for the prior distributions.Footnote ⁶ For the $\alpha $ parameters, setting the mean to zero is a natural choice (implying no shifting on average): $\alpha _i \sim \text {Normal}(0, \sigma _\alpha ^2)$ . In this specification, the standard deviation, $\sigma _\alpha $ , is an aspect of the population distribution to be estimated jointly with the other parameters. For the $\beta $ parameters, using a distribution with a median of one yields a latent scale with units similar to the observed one, and we can achieve this by using a log-normal distribution: $\beta _i \sim \text {Log-normal}(0, \sigma _\beta ^2)$ .Footnote ⁷

The use of a log-normal distribution also has a second motivation: It keeps the $\beta $ parameters away from zero.Footnote ⁸ As noted above, Hare et al. (Reference Hare, Armstrong, Bakker, Carroll and Poole2015) scaled respondents’ self-placements as shown in Equation (2) for each posterior draw, and this may lead to division by $\beta $ values that are arbitrarily close to zero—which in turn yields extreme and implausible draws for the latent respondent positions. This problem implies that not all relevant information has been incorporated in the model, and using a prior that keeps the $\beta $ parameters away from zero is one way of including more such information.

Another key point is that the reported self-placements represent additional data that inevitably contain errors. Instead of rescaling the self-placements while assuming they are measured without error, we can model these data probabilistically, making them part of the likelihood function. If we model the self-placements the same way as the stimuli placements, we get the following likelihood for these data:

(3) $$ \begin{align} \prod^{N}_{i=1}\phi(V_{i}|\alpha_i + \beta_i \chi_i, \sigma_{i}^2), \end{align} $$

where $\chi _i$ is a parameter representing the latent position of respondent i and $\sigma _{i}^2$ is a variance term.Footnote ⁹ The complete likelihood for the model is then the product of Equations (1) and (3).

Finally, we do have some information about what values are plausible for the latent positions: As they come from the same population, there is a limit to how extreme we would expect any respondent to be, relative to the rest. By treating the latent respondent positions as parameters to be estimated, we can model these hierarchically and let the degree of shrinkage be determined by the data: $\chi _i \sim \text {Normal}(0, \sigma _\chi ^2)\label {eq:vposprior}$ .Footnote ¹⁰ This results in a more nuanced posterior distribution where areas of the parameter space that yield extreme and implausible values for $\chi _i$ are weighted down compared to the BAM specification. Together, these adjustments should reduce the model’s sensitivity to noise and ensure meaningful results for all respondents.

2.3 A Hierarchical Mixture Model (HBAM)

The model outlined above restricts the stretch parameters to be positive, but as Aldrich and McKelvey (Reference Aldrich and McKelvey1977) noted, some respondents may flip the scale and thus have negative parameters. These respondents would see the ideological space as a mirror image, and report conservative actors as liberal, and vice versa. This possibility is one of the reasons why Aldrich and McKelvey did not attempt to restrict the stretch parameters to be positive, and later applications of their method suggest there is a notable minority of such respondents in a typical survey sample (e.g., Lo et al. Reference Lo, Proksch and Gschwend2014). I therefore expand the HBAM₀ model to allow for such behavior.

I consider each respondent’s potential scale flipping as a latent discrete parameter and modify the HBAM₀ model accordingly. I define the likelihood for each respondent as a mixture of two distributions, representing the flipped and non-flipped states. I further marginalize out the latent discrete flipping parameter, obtaining the following likelihood for $Y_{ij}$ , where I estimate the flipping parameter’s expectation, $\lambda _i$ :

(4) $$ \begin{align} \prod^{N}_{i=1}\prod^{J}_{j=1}[\lambda_i \phi(Y_{ij}|\alpha^*_{1i} &+ \beta^*_{1i}\theta_j , \sigma_{ij}^2) \nonumber\\ + \, (1- \lambda_i) \phi(Y_{ij}| \alpha^*_{2i} &+ \beta^*_{2i}\theta_j , \sigma_{ij}^2)]. \end{align} $$

In this specification, each respondent is given separate $\alpha $ and $\beta $ parameters for each state, denoted $\alpha ^*$ and $\beta ^*$ . The first set of these $\beta ^*$ parameters, $\beta ^*_{1i}$ , takes positive values, whereas the second set, $\beta ^*_{2i}$ , takes negative values. The $\alpha ^*$ and $\beta ^*$ parameters are given the same priors and hyperpriors as the $\alpha $ and $\beta $ parameters in the HBAM₀ model, with the exception that the prior on $\beta ^*_{2i}$ has a negative sign. The mixing proportion, $\lambda _i$ , is given a beta prior, specified in terms of its expectation, $\psi $ , and concentration, $\delta $ : $ \lambda _i \sim \text {Beta}(\psi \delta , (1 - \psi )\delta )$ .Footnote ¹¹ The expectation parameter is given the following prior: $\psi \sim \text {Beta}(8.5, 1.5)$ , which implies a 15% prior probability that a respondent flips the scale.Footnote ¹² This prior permits all plausible values for $\psi $ while emphasizing those that are more probable in light of existing studies.Footnote ¹³

Still treating the self-placements as data to be modeled probabilistically and the respondent positions as latent parameters, I also expand the likelihood for $V_i$ in Equation (3) to allow for flipping:

(5) $$ \begin{align} \prod^{N}_{i=1}[\lambda_i \phi(V_{i}|\alpha^*_{1i} &+ \beta^*_{1i}\chi^*_{1i} , \sigma_{i}^2) \nonumber\\ + \, (1- \lambda_i) \phi(V_{i}| \alpha^*_{2i} &+ \beta^*_{2i}\chi^*_{2i}, \sigma_{i}^2)]. \end{align} $$

The latent positions are still modeled as coming from the same population distribution: $\chi ^*_{1i}, \chi ^*_{2i} \sim \text {Normal}(0, \sigma _{\chi ^*}^2)$ , and the prior for $\sigma _{\chi ^*}^2$ is the same as for $\sigma _{\chi }^2$ .

To generate posterior draws for the latent discrete flipping parameters, $\kappa _i$ , I use their respective expectations, $\lambda _i$ : $\kappa _i \sim \text {Bernoulli}(\lambda _i)$ . I then use these draws to combine $\chi ^*_{1i}$ and $\chi ^*_{2i}$ and obtain draws for each respondent’s position, $\chi _i$ :

(6) $$ \begin{align} \chi_i = \kappa_i \chi^*_{1i} + (1 - \kappa_i) \chi^*_{2i}. \end{align} $$

Draws for the $\alpha $ and $\beta $ parameters are obtained the same way:

(7) $$ \begin{align} \alpha_i = \kappa_i \alpha^*_{1i} + (1 - \kappa_i) \alpha^*_{2i}, \end{align} $$

(8) $$ \begin{align} \beta_i = \kappa_i \beta^*_{1i} + (1 - \kappa_i) \beta^*_{2i}. \end{align} $$

To see the implications of the HBAM model and some of its key priors, it is helpful to simulate draws from the prior for $\beta _i$ , integrating over the relevant hyperpriors. Figure 1 summarizes the draws from such a simulation. The prior distribution for $\beta _i$ is a bimodal mixture, reflecting the two possibilities that respondents either do or do not flip the scale. The medians of the two mixture components are set to 1 and $-1$ to identify the latent space, whereas the concentration of each component will be determined by the data, due to the hierarchical prior structure. The positive part of the distribution contains more probability mass than the negative part, which reflects the expectation that scale flipping is less likely than non-flipping. Again, the relative weights of the negative and positive components of the mixture will predominantly be determined by the data, given the hierarchical setup. A more comprehensive discussion of the priors is provided in the Supplementary Material (including prior simulations for all parameters).

Figure 1 Prior simulation for $\beta _i$ in the HBAM model, integrating over all relevant hyperpriors.

2.4 Specification of Errors

Implementing the models in Stan makes it natural to model the standard deviation of the errors, $\sigma _{ij}$ , more directly than one would in JAGS—where the use of non-conjugate priors leads to inefficient Metropolis sampling. Specifically, $\sigma _{ij}$ is constructed as a function of two parameters: $\sigma _{ij} = \rho _j \sqrt {\eta _{i}}$ , where $\eta _i$ captures the average variance of respondent i (implicitly increased by a factor of $J^2$ for more efficient sampling), whereas $\rho $ is a unit J-simplex vector, splitting the variance by stimuli. The simplex vector is given a weakly informative, symmetric Dirichlet prior: $\rho \sim \mathrm {Dirichlet}(5)$ , whereas $\eta _{i}$ is given a scaled inverse chi-squared prior, $\eta _{i} \sim \text {Scale-inv-}\chi ^2(\nu ,J^2 \tau ^2)$ .Footnote ¹⁴ To ensure that the models are comparable and sampling efficiently, this specification is used for all models—including the BAM model.Footnote ¹⁵

3.1 K-fold Cross-Validation

I start by estimating the pointwise out-of-sample prediction accuracy of the models, which will help assess model fit and overfitting. Unpooled models are prone to overfitting, and we would expect the HBAM model to outperform the BAM model. A comparison between the HBAM₀ and HBAM models will also provide information on whether allowing for scale flipping is worthwhile—or an unnecessary complication.

To compare the models, I perform a 10-fold cross-validation on the reported stimuli positions.Footnote ¹⁹ In other words, I randomly partition the data into 10 equally sized subsets (using stratified sampling so that each respondent at most provides one observation to each subset). I then fit the models 10 times, each time holding out one of the subsets, while evaluating the pointwise log likelihood of the held-out data given the estimated parameter values. For each model, I then estimate the expected log pointwise predictive density (ELPD) and its standard error.

The results of the cross-validation are shown in Table 1. We see that the estimated ELPD is significantly lower for the HBAM₀ model than for the BAM model, while the HBAM model has a significantly higher estimate than either of the other models. In short, the results suggest that the HBAM model performs better than the other models in terms of predicting new data. The lower performance of the unpooled model is expected and implies that this model is overfitting the data. The difference between HBAM₀ and the other models implies that the HBAM₀ model is too restrictive and underfits the data: Allowing for scale flipping yields a considerable improvement in predictive accuracy.

Table 1 Estimated ELPDs based on a 10-fold cross-validation. ELPD is the theoretical expected log pointwise predictive density for a new dataset, whereas $\mathrm{SE}_{\widehat {\mathrm{ELPD}}}$ is the standard error of the ELPD estimate.

3.2 Estimated Shifting

One of the key findings of Hare et al. (Reference Hare, Armstrong, Bakker, Carroll and Poole2015) was that respondents tend to shift the ideological space in a way that may lead us to underestimate polarization: They move the stimuli that they disagree with too far away on the other side of the scale while understating their own ideological extremism. This pattern is reproduced in Figure 2, which shows how the estimated $\alpha $ parameters from the BAM and HBAM models are distributed over respondents’ self-placements. Those on the left tend to shift their placements to the right, whereas those on the right tend to shift their placements to the left. The two models produce a similar picture, but the estimates from the BAM model are larger and contain more outliers.Footnote ²⁰

Figure 2 Estimated shifting over self-placements. The plots summarize posterior median $\alpha _i$ estimates.

Figure 3 Estimated stretching over self-placements. The plots summarize medians of absolute values of posterior draws for $\beta _i$ .

3.3 Estimated Stretching

The fundamental logic behind these models implies that we would expect to find a particular pattern in the estimated $\beta $ parameters: A key assumption is that some respondents stretch the space more than others, and thus place both the stimuli and themselves at more extreme positions. In short, we would expect to find that those who place themselves at more extreme positions tend to stretch the space to a greater extent. This is why the models use the stimuli placements to estimate the degree of stretching and rescale the respondents’ self-placements accordingly. If the mentioned assumption does not hold, a key part of these models is invalidated.

As the $\beta $ parameters can take on both positive and negative values in both the BAM and HBAM models, the most straightforward way to gauge the overall degree of stretching regardless of flipping is to use the median of the absolute values of the posterior draws for each respondent. The boxplots in Figure 3 show these median absolute $\beta $ estimates from each model over respondents’ self-placements. Interestingly, the $\beta $ parameter estimates produced by the BAM model follow a virtually flat line across all but the center category. In contrast, the estimates from the HBAM model follow a V-shape, where the $\beta $ estimates are more extreme for respondents who place themselves at the extremes of the scale. This is important because it is exactly the kind of pattern we would expect to find if the assumption behind these models is correct. It also illustrates that the models can yield substantively different conclusions, even at an aggregate level.

3.4 Estimated Respondent Positions

A key problem with the current implementation of the BAM model is that it does not yield meaningful marginal posterior distributions for all respondent positions. To rescale a respondent position, the BAM model subtracts the draws for $\alpha _i$ and divides by the draws for $\beta _i$ , which can result in extreme values if the draws for $\beta _i$ get close to zero. This problem is illustrated in Figure 4, which focuses on one of the respondents for which the BAM model produces a problematic distribution. Along with the draws for the respondent’s position, the plot shows the estimated population distribution (as the density of the draws for all respondents).

Figure 4 Posterior distribution for the position of a specific respondent compared to the population. The gray areas summarize the draws for one of the respondents for which the BAM model produces a problematic distribution. The black lines show the density of the draws for all respondents. The horizontal axes have been capped at $\pm $ 10, although the draws produced by the unpooled model range from below $-$ 10,000 to above 10,000.

From a Bayesian perspective, the population distribution contains relevant information about the positions of individual respondents: If we had no other information, the population distribution would define a broad range within which we would expect a randomly chosen respondent to be located. In Figure 4, most of the BAM model's draws for the selected respondent are in areas that are highly implausible according to the population distribution. At the most extreme, the draws go beyond $\pm $ 10,000. In contrast, the HBAM₀ and HBAM models avoid this problem by treating the latent respondent positions as parameters and incorporating information about the population distribution in their priors.

To assess the extent of this issue, I compare all individual respondents’ marginal posterior distributions to the estimated population distribution. For each model, I calculate the range within which 95% of the draws for all respondent positions lie. This is an approximation of where we would expect most respondents to be located, and if individual respondents have 95% credible intervals (CIs) notably wider than this range, it suggests that their posterior distributions are too wide. I therefore calculate the 95% CI for each respondent’s position and compare these to the population distribution. For the BAM model, the share of CIs that are more than 50% wider than the population 95% range is 8%. For the HBAM₀ and HBAM models, the number of such cases is zero.

In addition to the problem shown in Figure 4, we would expect the overfitting demonstrated by the cross-validation to reduce the accuracy of the BAM model’s point estimates of the respondent positions. Figure 5 shows the estimated posterior median respondent positions over respondents’ self-placements. The most notable about the results from the BAM model is the amount of extreme respondent position estimates (compared, e.g., to the interquartile range represented by the boxes). Even respondents who place themselves at zero are occasionally moved to extreme positions. In contrast, the HBAM results show a pattern where respondents are rarely moved to much more extreme positions than they place themselves at, although they may be flipped to the other side.

Figure 5 Respondent position estimates over self-placements. The estimates are posterior medians.

Figure 6 shows the posterior median respondent position estimates from the HBAM model over the estimates from the BAM model. Again, we see that the BAM model produces some very extreme estimates, even for respondents that are considered centrist by the HBAM model. The correlation between the two sets of estimates is .6, which implies a substantive difference in the results from the two models. To see whether this also reflects a difference in the models’ abilities to uncover the true latent parameters, I conduct a Monte Carlo study in the next section.

Figure 6 Respondent position estimates from the HBAM model over estimates from the BAM model. The estimates are posterior medians.

3.5 Estimated Hyperparameters

Table 2 summarizes the marginal posterior distributions of the hyperparameters in HBAM model. Notably, $\psi $ is estimated to be between .88 and .90 with 95% probability. This implies a 10%–12% probability that an individual respondent flips the scale, which is plausible in light of previous studies (e.g., Lo et al. Reference Lo, Proksch and Gschwend2014). The other estimates are less substantively interesting, but relevant for the Monte Carlo study below.

Table 2 Summary of marginal posterior distributions for the hyperparameters in the HBAM model.

4.1 Setup

Because we would expect model performance to depend on the amount of data available at the individual level, I run simulations for different numbers of stimuli. In Lo et al.’s (Reference Lo, Proksch and Gschwend2014) study of 22 countries, the number of stimuli ranged from 5 to 11, with a mode of 8, and I run simulations for $J \in \{4, 8, 12\}$ . Lo et al. also found nontrivial shares of negative stretch parameters—typically around 5%–10%. To see how the models perform with and without a moderate degree of scale flipping, I run simulations for $\psi \in \{.85, 1\}$ . Finally, to see how robust the models are to random noise, I incrementally increase the scale of the errors, $\tau $ , across a wide range covering all commonly observed values.Footnote ²¹

To make the simulated data as realistic as possible, the remaining parameter values for the data generating process are taken from the empirical application reported above (see Table 2). For each combination of J and $\psi $ values, I simulate 1,000 datasets with $N = 500$ . I then fit the three models to each set and calculate relevant performance criteria.Footnote ²² Because the models yield differently scaled results, measures like the root-mean-square error are not directly applicable without some normalization, and I focus instead on the correlation (Pearson’s r) between the sets of estimated and true respondent positions.