3.1 Setup

Suppose there are J items that respondents rank. Here, we assume that (1) all respondents have well-defined preferences (completeness and transitivity) and (2) respondents rank all items. We also assume that each person has two potential ranking responses: non-random and random responses. First, we define non-random responses as responses based on underlying preferences or intentions. Non-random responses do not need to perfectly align with people’s underlying preferences as long as they reflect respondents’ substantial intent, whether sincere or strategic.

Next, we define random responses as meaningless responses that are independent of people’s preferences—irrelevant to/unreflective of true preferences. Our definition is general, and random responses can take many forms. For example, Figure 2a visualizes what occurs when respondents provide random response $(1, 2, 3, 4)$ in four commonly used ranking question formats. Here, random responders (i) use the same order as a presented list (drag and drop), (ii) draw a diagonal line from the top-left to the bottom-right (radio buttons), (iii) enter in numerical ascension when asked to rank (numeric entry), or (iv) do not reorder a presented item (select box). Many other patterns are also possible. Figure 2b provides three visually intuitive examples of random responses we call diagonalizing, zigzagging, and dog-legging in the radio-button format (for a similar discussion in RCV, see Atsusaka Reference Atsusaka2025).

Figure 2 Examples of random responses in ranking questions.

Let $Y^{*}_{i}$ and $e_{i}$ be respondent i’s ( $i=1, \ldots ,N$ ) non-random and random responses (or errors), respectively. Let $z_{i}$ be a random variable denoting whether respondent i offers a non-random response ( $z_{i}=1$ ) or otherwise ( $z_{i}=0$ ). We denote respondent i’s observed response by $Y^{\text {obs}}_{i} = Y^{*}_{i}z_{i} + e_{i}(1-z_{i})$ . We use a general notation $g(\cdot )$ to represent a ranking-based quantity of interest (QOI).Footnote ⁶

Our framework represents the observed ranking data as a mixture of non-random and random responses:

(1)

where $\Pr (z_{i}=1)$ and $\Pr (z_{i}=0)$ are the proportions of non-random and random responses, respectively.

This illustrates why researchers cannot simply use raw data to study their variable of interest—the data contains the information on what they wish to analyze and irrelevant noises that skew their understanding of their target concept. Appendix A.1 of the Supplementary Material formally illustrates the bias from random responses and discusses why it is consequential in empirical studies. The fundamental problem of random responses is that there is no way for researchers to know which part of their data is susceptible to errors (i.e., which respondents offer random responses) and to what extent.

3.2 Quantities of Interest and Identification Problems

There are two classes of quantities that interest researchers, differing in terms of the population of interest in which the target quantity is defined. The following distinction relates to the difference between the average engaged response among the engaged and the average engaged response discussed in Tyler et al. (Reference Tyler and Westwood2024).

The first quantity is a ranking-based quantity among non-random responses:

(2) $$ \begin{align} \theta_{z} \equiv g(Y^{*}_{i}|z_{i}=1). \end{align} $$

This paper mainly studies the identification of $\theta _{z}$ . We highlight that this estimand is only defined among people who offer non-random responses. Rearranging Equation (1), our identification problem becomes

(3) $$ \begin{align} \underbrace{g(Y^{*}_{i}|z_{i}=1)}_{\text{what we wish to study}} = \frac{\overbrace{g(Y^{\text{obs}}_{i})}^{\text{derived from raw data}} - \overbrace{\Pr(z_{i}=0)}^{\text{prop. random resp.}}\overbrace{g(e_{i}|z_{i}=0)}^{\text{random resp.}}}{\underbrace{1 - \Pr(z_{i}=0)}_{\text{prop. non-random resp.}}}. \end{align} $$

The right-hand side contains three quantities: (1) the target quantity based on observed rankings $g(Y^{\text {obs}}_{i})$ , (2) the proportion of random responses $\Pr (z_{i}=0)$ , and (3) the target quantity based on random responses $g(e_{i}|z_{i}=0)$ .

The key problem is that we only observe the first quantity $g(Y^{\text {obs}}_{i})$ . Thus, without making any assumptions, the QOI will never be estimated from raw data alone, regardless of how many responses researchers collect. Section 4 discusses how our design-based methods allow us to point-estimate the latter two unknowns.

The second quantity of interest is a ranking-based quantity in the target population from which samples are drawn:

(4) $$ \begin{align} \theta & \equiv g(Y^{*}_{i})\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\quad \end{align} $$

(5) $$ \begin{align} & = \underbrace{g(Y^{*}_{i}|z_{i}=1)}_{\theta_z}\Pr(z_{i}=1) + \underbrace{g(Y^{*}_{i}|z_{i}=0)}_{\text{counterfactual}}\Pr(z_{i}=0), \end{align} $$

where, $\theta $ becomes closer to $\theta _{z}$ as the probability of error-free responses $\Pr (z_{i}=1)$ increases.

Generally, $\theta $ is more difficult to identify than $\theta _z$ because it requires an additional assumption about the counterfactual quantity $g(Y^{*}_{i}|z_{i}=0)$ —non-random rankings that random respondents would have provided had they not responded randomly. In Section 6, we discuss three identification strategies for this population-level quantity.Footnote ⁷

4.1 Overview

Figure 3 summarizes our design-based methodology, which leverages two survey designs: item order randomization and an anchor question.

Figure 3 Design-based methods for estimating the proportion and distribution of random responses.

4.2 Item Order Randomization

The first design consideration is item order randomization. Many survey platforms support randomization, and some studies use item order randomization with ranking questions (Costa Reference Costa2021; Malhotra and Margalit Reference Malhotra and Margalit2014; Pradel et al. Reference Pradel, Zilinsky, Kosmidis and Theocharis2024; Rathbun, Rathbun, and Pomeroy Reference Rathbun, Rathbun and Pomeroy2022). The primary role of item order randomization is to identify the distribution of rankings among random responses and the direction of the bias with respect to our quantities of interest.

Our key theoretical result is that, under item order randomization, the rankings among random responses follow a uniform distribution with probability $\frac {1}{J!}$ , where J is the number of items.Footnote ⁸ For example, with three items, random responses will correspond to one of the six profiles in the set $\{123, 132, 213, 231, 312, 321\}$ with probability $\frac {1}{6}$ .Footnote ⁹

Let $U_{J}$ be a random variable that follows a discrete uniform distribution with J ranking items. Then, our result implies that

(6) $$ \begin{align} g(e_{i}|z_{i}=0) & = g(U_{J}). \end{align} $$

This result is powerful because it holds regardless of the pattern of response patterns—item order randomization transforms all random responses into a set of rankings following the uniform distribution.

Our result also clarifies why the bias still remains even after randomization. Integrating Equations 1 and 6, we can show that observed data still contain random responses under randomization as follows:

(7)

The above equation shows that, under randomization, random responses will pull any estimates towards what researchers may observe when all respondents are indifferent among available items. Thus, even under randomization, random responses still affect researchers’ substantive inferences by “diluting” their conclusions.

4.3 Anchor Questions

To identify the proportion of random responses at the time of the target ranking question, $\text {Pr}(z_{i}=0)$ , we propose using an auxiliary ranking question whose “correct answer” is ex-ante known to researchers (for a similar idea, see Atsusaka and Stevenson (Reference Atsusaka and Stevenson2023)). We call this an anchor question and ask it right before or after the target ranking question.Footnote ¹⁰ The item order in the anchor question must be randomized just as it must be randomized for the target question. To illustrate, Figure 4 presents the anchor question we included in our survey. In this example, the question asks respondents to rank four communities from the smallest to the largest, and the correct answer is (household, neighborhood, city or town, state).Footnote ¹¹

Figure 4 Example of an anchor ranking question.

Estimating the Proportion of Random Responses in the Anchor Question

First, we estimate the proportion of random answers in the anchor question by using correct responses. Let $c_{i}$ be a binary variable indicating whether respondent i offers the correct answer in the anchor question $(c_{i} = 1)$ or not $(c_{i} = 0)$ . Let $\Pr (z^{\text {anc}}_{i}=0)$ and $\Pr (z^{\text {anc}}_{i}=1)$ be the proportions of random and non-random responses in the anchor question, respectively.Footnote ¹² Under item order randomization, we can estimate the proportion of random responses in the anchor question using the following estimator.

Proposition 1 Unbiased Estimator of the Proportion of Random Responses in the Anchor Question

(8) $$ \begin{align} \widehat{\Pr}(z^{\text{anc}}_{i}=0) & = 1 - \widehat{\Pr}(z^{\text{anc}}_{i}=1) \qquad\qquad\qquad\qquad\qquad\qquad \end{align} $$

(9)

The proof is in Appendix A.3 of the Supplementary Material. Equation 9 has intuitive interpretations. The second term suggests that the proportion of non-random responses can be estimated from the proportion of correct answers $\frac {\sum _{i=1}^{N}c_{i}}{N}$ after accounting for the probability that random responses happen to be correct (which is $\frac {1}{J!}$ under item order randomization).Footnote ¹³ The third term can then be interpreted as renormalization to ensure that the resulting quantity becomes probability.

4.4 Bias-Corrected Estimators

Integrating the above results, we propose two approaches to correct measurement errors. The first strategy is to directly correct the bias with a specific quantity of interest. The second approach is to apply the idea of inverse probability weighting (IPW).

4.4.1 Direct Correction

The first approach is to use the following bias-corrected estimator:

(10)

This estimator is simple and only requires one extra estimation compared to the naïve estimator while retaining the original sample size. Our proposed estimator has a wider confidence interval than the naïve estimator due to the additional uncertainty around the estimated proportion of correct answers. We use bootstrapping for constructing confidence intervals. Moreover, researchers can include survey weights in $\widehat {g}(Y^{\text {obs}}_{i})$ as in typical survey data analysis, where survey weights represent the product of the design weight and a poststratification or calibration adjustment.

4.4.2 Inverse Probability Weighting

The second strategy is to leverage the idea of IPW. Under this framework, the problem of measurement errors (Equation 1) can be considered an issue of selection bias. Figure 5 illustrates this idea graphically. Here, due to random responses, relatively popular rankings (e.g., 123) are under-sampled, while relatively unpopular rankings (e.g., 231) are over-sampled compared to their true distribution. A natural solution is to weight up a set of rankings that are supposed to be more prevalent and weight down a set of rankings that are supposed to be less prevalent than what raw data suggest.

Figure 5 Graphical representation of IPW.

Let $\mathbb {P}(Y^{*}_{i}|z_{i}=1)$ be the population proportion of respondent i’s ranking profile given non-random responses. Let $\mathbb {P}(Y^{\text {obs}}_{i})$ be the same proportion based on observed data, and $w = \{w_{i}\}_{i=1}^{N}$ be a vector of weights for N respondents included in nonparametric or parametric analyses. We propose the following inverse probability weight:

(11) $$ \begin{align} w_{i} = \Bigg[\frac{\mathbb{P}(Y^{\text{obs}}_{i})}{\mathbb{P}(Y_{i}^{*}|z_{i}=1)} \Bigg]^{-1}. \end{align} $$

The weights can be nonparametrically identified via the following plug-in estimator:

(12) $$ \begin{align} \widehat{w}_{i} = \Bigg[\frac{\widehat{\mathbb{P}}(Y^{\text{obs}}_{i})}{\widehat{\mathbb{P}}(Y_{i}^{*}|z_{i}=1)}\Bigg]^{-1}. \end{align} $$

Let $\mathbb {P}(U_{J})$ be the uniform distribution with probability $\frac {1}{J!}$ . Building on a similar derivation to Equation A.7, the denominator can be unbiasedly estimated with the following estimator:

(13)

Researchers can also use survey weights in the IPW framework by constructing a new weight $w^{*}_{i} = w_{i}w_{i}^{s},$ where $w_{i}^{s}$ is respondent i’s survey weight.

4.4.3 Methods Selection

The two approaches complement each other. Table 1 provides a comparison. We recommend that researchers use the direct approach whenever their target quantities are simple and nonparametrically identifiable (e.g., average ranks) as it provides exact bias correction to their QOIs. In contrast, when they wish to perform more complex and parametric analyses, such as running regressions, the IPW framework can be helpful, as it allows researchers to perform any analyses with the bias-correction weights.

Table 1 Comparison of two bias correction methods.

4.5 Uniformity Test

Finally, our framework also allows researchers to detect the presence of random responses without requiring any anchor questions. Appendix B of the Supplementary Material introduces the uniformity test, which shows that recorded responses (what respondents submit in Figure 2; see Appendix A.2 of the Supplementary Material) will follow a uniform distribution in the absence of random responses. Conversely, non-uniformity in the test suggests the presence of random responses.

5.1 Analysis of the Anchor Question

First, we estimate the proportion of random responses using our anchor question. Table 2 reports the number of each ranking response to the anchor question. We code 1234 (household < neighborhood < city or town < state) as the correct response ( $c_{i}=1$ ) and other rankings as the incorrect response ( $c_{i}=0$ ). The result shows that the empirical proportion of correct responses is $\frac {\sum _{i=1}^{N}c_{i}}{N} = \frac {754}{1082} \approx 0.697$ . Proposition 1 states that the estimated proportion of random responses can be estimated as $1 - [\frac {754}{1082} - \frac {1}{24}](1 - \frac {1}{24})^{-1} \approx 0.316$ . That is, we find that about 31.6% of ranking answers are random responses in the anchor question.Footnote ¹⁶

Table 2 Distribution of responses to the anchor question.

Note: 1234 corresponds to household $\rightarrow $ neighborhood $\rightarrow $ city or town $\rightarrow $ state, which is coded as the correct anchor response.

By invoking Assumption 1, we estimate that 31.6% of respondents offer random responses in our target ranking question. We believe that this is not an unreasonably high (or low) estimate. For example, Berinsky, Margolis, and Sances (Reference Berinsky, Margolis and Sances2014), using four different screeners, show that the failure rates for them ranges from 34% to 41%. Relatedly, Atsusaka (Reference Atsusaka2025) finds that about 31% and 37% of respondents in survey experiments offered the same ranking response to two different ranking questions. Moreover, Clayton et al. (Reference Clayton, Horiuchi, Kaufman, King and Komisarchik2023) study measurement errors in conjoint experiments and estimate that about 19.3–27.0% of respondents offered different responses to an identical question that was asked twice.

In some cases, researchers may encounter “debatable cases,” in which multiple responses can be considered correct even after carefully designing anchor questions. For example, based on substantive knowledge, some researchers may think that for some people, the size of the community should be ordered as household < neighborhood < state < city or town. In another example, after analyzing data, analysts may find that some ranking responses are more prevalent than other incorrect answers (e.g., 4321 in Table 2). In these cases, analysts can code more than one response as correct answers. Moreover, researchers can also give credit to “partially correct answers,” if any, by coding such responses with known probability (e.g., coding an 80% correct response as correct with probability 0.8). Furthermore, it is also possible to use the most conservative (only one correct answer) and most liberal (as many correct answers as possible) coding schemes to make bounds for the resulting estimates.

Note that conservative (liberal) coding leans toward an over-estimation (under-estimation) of the prevalence of random responses in the anchor question. More importantly, the main focus should be on satisfying Assumption 1 when researchers consider different coding schemes for $c_{i}$ . If anything, we recommend underestimating rather than overestimating the proportion of random responses in the target question because it leads to under-correction of the bias, which guards against inflating Type I errors. For example, if researchers suspect that there are more random responses in the anchor than in the main question (e.g., the anchor looks like an attention check, which caused more respondents to answer randomly), it would be better to code fewer anchor responses as correct answers should the coding is debatable.

5.2 Summarizing Data with Empirical Distributions

We begin our analysis by describing the distribution of our data. The left panel of Figure 7 presents the distribution of all possible rankings with our methods, with the gray region indicating where party was ranked first. We find that while a great variation in the ranking outcome exists, people rarely rank political party as their first choice. This may provide evidence that relative partisanship is rather low among American adults—a notable finding given the emphasis on partisan identity in American political behavior. We also identify that three rankings/orderings are particularly prevalent, including (gender, race, religion, party), (gender, race, party, religion), and (religion, gender, race, party).

Figure 7 Distributions of identity rankings with bias-corrected and raw data.

The right panel of Figure 7 visualizes what researchers could have observed had our methods not been applied (but item order randomization is still implemented). Here, many unpopular rankings (e.g., those starting with party) are overrepresented due to random responses. Indeed, the panel leads to a different conclusion that the relative importance of party is as much as that of race. Again, this demonstrates that random rankings, under item order randomization, pull the naïve estimates towards uniformity, where each ranking profile is equally prevalent.

5.3 Understanding Average Patterns

Next, we study the average ranks of the four items as another way to measure relative partisanship. Figure 8 visually compares the results based on our methods and raw data. Overall, we find that the average rank is the lowest for political party, followed by religion, race and ethnicity, and gender. Consistent with our statistical argument (Appendix A.1 of the Supplementary Material), the difference between bias-corrected and unadjusted estimates (and thus bias) is larger when the unknown target parameter is farther away from the average rank based on uniformity (in this case, $\frac {1+4}{2} = 2.5$ ).

Figure 8 Average ranks with and without bias correction

Note: The dashed line represents the average rank that arises when people are indifferent among the four items.

For party and gender, bias-corrected estimates are statistically significantly different from unadjusted estimates and closer to their bound values (1 and 4). Accordingly, the difference between religion and party is 1.52 (direct) and 1.36 (IPW) times larger in our methods than unadjusted estimates. Similarly, the difference between party and gender is 1.55 (direct) and 1.41 (IPW) times larger in our estimates than unadjusted estimates. In contrast, bias-corrected and raw-data estimates are similar for race and religion. This is consistent with our argument because while bias pulls the estimated average ranks of race and religion towards 2.5, unadjusted estimates of the two items were already close to the value. This illustrates that the magnitude of the bias and the difference between bias-corrected and unadjusted estimates varies not only by the proportion of random responses but also by the values of the target parameters. Thus, researchers should keep in mind that finding a small difference for a particular item after bias correction does not mean that the methods “failed” to address measurement errors.

Researchers can also estimate many other quantities of interest while applying bias correction. For example, our software, rankingQ , supports the pairwise ranking probability for items j and $j^{\prime } \Pr (Y_{ij} < Y_{ij^{\prime }})$ , the top-k ranking probability $\Pr (Y_{ij} \leq k)$ , and the marginal ranking probability $\Pr (Y_{ij} = k)$ in addition to the average rank $\mathbb {E}[Y_{ij}]$ .

5.4 Regression and Predicted Probability

Moreover, we analyze how respondent characteristics influence their relative partisanship while applying bias correction. To do so, we construct a Plackett–Luce model (also known as rank-order logistic regression) to associate people’s ranking choices with their attributes (Alvo and Yu Reference Alvo and Yu2014; Train Reference Train2003). We regress identity rankings on age, gender, race, education level, ideology, partisanship, and region, while incorporating survey and bias-correction weights via the IPW framework. After estimation, we generate the predicted probabilities that people submit a particular ranking profile with 95% confidence intervals via parametric bootstrapping (Tomz, Wittenberg, and King Reference Tomz, Wittenberg and King2003) over the range of the ideology variable (7-point scale).

Here, we examine how ideology influences relative partisanship among Americans who are 40 years old, white, male, independent, with some college education, and living in the Northeast. We examine four ranking profiles, including the three most prevalent rankings discussed in Section 5.2 and the most prevalent ranking profile starting with party.

Figure 9 presents our results. The top-left panel shows that people are more likely to choose (gender, race, party, religion) as they become more liberal, all things being equal. The first difference in predicted probabilities between the most liberal and conservative Americans with bias correction is roughly $0.313 - 0.067 = 0.246$ , which is almost 1.6 times larger than its unadjusted counterpart of $0.203 - 0.046 = 0.157$ . This illustrates how random responses can weaken the association between an independent variable and a target ranking profile.

Figure 9 Predicted probabilities with and without bias correction.

The top-right panel suggests that ideology only weakly relates to ranking (gender, race, religion, party). While both results show similar patterns, on average, bias-corrected predictions (0.175) are 0.077 points higher than unadjusted predictions (0.109). Thus, without correction, researchers can underestimate the prevalence of the target ranking profile by 1.61 times. Finally, the lower two panels provide examples of relatively similar bias-corrected and unadjusted predictions. Importantly, this does not mean that our methods “did not work.” Rather, it illustrates that the nature of bias depends on the target ranking profile, the target independent variable, and the reference values at which other variables are fixed, in addition to the proportion of random responses.

6.1 Comparison with Alternative Designs

Listwise Deletion

Some readers may wonder how our methods differ from more traditional solutions relying on attention checks, repeated questions, and so on. More specifically, how is our proposal different from listwise deletion based on these alternative design considerations?Footnote ¹⁷

Let $z_{i}^*$ be a binary variable taking 1 if respondent i passes a certain instructional manipulation check and 0 otherwise. For example, $z_{i}^* = 1$ when respondent i passes an attention check, provides the same answer to the same question asked multiple times, or does not speed through the target question. Researchers then drop all respondents who did not pass the test (i.e., delete all i if $z_{i}^* = 0$ ) and produce “cleaned” data of size $N_c, \{Y_{i}^{\text {obs}}(z_{i}^* = 1)\}_{i=1}^{N_c}$ .

When adopting this strategy, researchers often implicitly assume the following.

Assumption 2 Individually Constant Randomness

Random responders in the target ranking question are identical to those who fail the instructional manipulation check. Formally, $z_{i} = z^{*}_{i}$ for all $i = 1, \ldots , N$ .

Invoking Assumption 2, listwise deletion identifies $\theta _{z}$ as follows:

(14) $$ \begin{align} g(Y^{\text{obs}}_{i}|z_{i}^* = 1) & = g(Y^{*}_{i}|z_{i}=1)\qquad \end{align} $$

(15) $$ \begin{align} & = \theta_{z}. \end{align} $$

This way, listwise deletion along with Assumption 2 allows researchers to estimate $\theta _z$ directly from the “cleaned” data. In other words, it is assumed that those who failed the test also provided random responses to the target ranking question. Importantly, this assumption requires that attention is stable for all respondents across the test and the target question. In this sense, Assumption 2 is much stronger than Assumption 1, which requires only the proportion of random responses to be the same.

Although the assumption is not directly verifiable, we collected auxiliary information to examine its plausibility in our survey. We find that Assumption 2 is indeed strong; as we show in Appendix C.3 of the Supplementary Material, even between two attention checks, there is very little correlation ( $\rho = 0.25$ ). This is why we propose anchor questions—to approximate the randomness in the target ranking question by using a similar ranking question asked right before or after it.

6.2 Identification of $\theta $

We now propose three identification strategies for $\theta = g(Y^{*}_{i}) = g(Y^{*}_{i}|z_{i}=1)\Pr (z_{i}=1) + g(Y^{*}_{i}|z_{i}=0)\Pr (z_{i}=0)$ —the ranking-based quantity among all people in the target population. The key is to identify $g(Y_{i}^{*}|z_{i}=0)$ , which is a function of counterfactual rankings that random respondents would have provided had they responded non-randomly.

The first approach is to assume that those who provide random responses are indifferent among available items. More specifically, we assume that the counterfactual ranking of J items is a uniformly distributed random variable $\mathcal {U}_{J}$ .

This assumption is plausible, for example, when respondents offer random responses because they do not have sufficient information about available options. Here, randomness and preference are correlated. For example, in RCV elections, voters with low education levels may be more likely to provide random responses and have uniform preferences as they have less contextual knowledge to rank multiple candidates.

With Assumption 3, it is straightforward to compute $g(Y^{*}_{i}|z_{i} = 1)$ using a uniform distribution and then estimate $\theta $ accordingly. However, our design-based methods provide an even simpler solution. Using item order randomization, we can show that

(16)

(17)

(18)

(19) $$ \begin{align} & = g(Y^{\text{obs}}_{i}).\qquad\qquad\qquad\qquad\qquad\qquad\qquad\quad \end{align} $$

In other words, $\theta $ can be estimated directly from raw data alone.

A second approach is to assume that random respondents would have submitted similar rankings to non-random respondents. More specifically, we assume the following.

This assumption is plausible, for example, when random responses are based on simple misunderstandings, confusions, or mistakes that prevent respondents from expressing their underlying preferences. We call this assumption contaminated sampling building on Horowitz and Manski (Reference Horowitz and Manski1995). With Assumption 4, researchers can identify $\theta $ by replacing counterfactual rankings with observed ones as follows:

(20)

(21)

(22) $$ \begin{align} & = g(Y^{*}_{i}|z_{i} = 1) \qquad\qquad\qquad\qquad\qquad \qquad\qquad\end{align} $$

(23) $$ \begin{align} & = \theta_{z}. \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\end{align} $$

Assumption 4 is violated whenever there exists a confounder that relates to both randomness and preference. Our final approach is to relax the assumption by conditioning on such a confounder. Let $\mathbf {X}_{i}$ be a set of covariates that are related to both random responding $z_{i}$ and preference $Y_{i}^*$ . We assume the following.

For simplicity, consider a single confounder. Let $\mathbf {x}$ be a specific covariate value and $\mathcal {X}$ be its sample space. Combined with Equation 23, we propose the following identification strategy via stratification:

(24)

(25)

In other words, we compute the weighted average of $\theta _{z}$ in each distinct category defined by the covariate, where the weight is the proportion of each stratum $\Pr (\mathbf {X}_{i}=\mathbf {x})$ . For example, suppose that strength in partisanship is related to both random responding and identity ranking. Then, researchers can estimate $\theta $ by estimating $\theta _z$ within groups of people who have reported the same partisan strength and then sum up the estimates while weighting them by the proportions of the groups.

To illustrate the three strategies, Figure 10 presents the estimates of $\theta $ (average rank) under the three different assumptions. We use partisan strength (Independent, Weak Partisan, Strong Partisan) to illustrate the stratification approach, which yields similar estimates to the contaminated sampling approach. The uniform preference approach yields estimates closer to $\frac {1 + 4}{2} = 2.5$ than the other two methods, consistently with its assumption. This way, researchers can extend their inference to $\theta $ by leveraging their substantive knowledge about why random responses may occur in their specific application.

Figure 10 Average ranks in the entire population under different assumptions.

Note: The dashed line represents the average rank that arises when people are indifferent among the four items.