1 Acquiescence-Response Bias

Acquiescence bias—“the tendency to endorse any assertion made in a question, regardless of its content” (Krosnick Reference Krosnick1999, 552)—inflates endorsement and agreement in survey response. Acquiescence bias has long been appreciated by psychologists and varies in magnitude across the population (e.g., Billiet and McClendon Reference Billiet and McClendon2000; Krosnick Reference Krosnick1999; Schuman and Presser Reference Schuman and Presser1981; Watson Reference Watson1992). Schuman and Presser (Reference Schuman and Presser1981, Chapter 8) summarize three interpretations of this heterogeneity. First, tendency to agree might be an individual personality trait generated by genes and environment. Second, acquiescence bias might follow from status differentials between participant and surveyor. Third, acquiescence bias might be a heuristic response rule for participants when they do not know how to respond to a question.

Because magnitude of acquiescence bias correlates with individual characteristics such as education, inferences about the relationship between political characteristics and beliefs might also be inaccurate in the presence of acquiescence bias. This problem, in fact, was appreciated by survey researchers in political science in the 1960s. The authors of The American Voter found that acquiescence bias undermined conclusions others had drawn about the psychological authoritarianism scale. They showed that the existing finding that those with less education were more authoritarian was due only to the higher rate of acquiescence bias among those with less education (Campbell et al. Reference Campbell, Converse, Miller and Stokes1960, 512–514).

Acquiescence bias is a common concern in psychometrics where researchers construct psychology indices based on responses to multiple Agree/Disagree Likert items. To improve measurement and validity, psychologists field multiple items thought related to the underlying index. But, if all items are coded such that the agree response indicates one end of the index and disagree the other, then the subset of respondents who exhibit acquiescence bias will be improperly placed toward the agree end of the index.Footnote ³

Psychometricians use a variety of techniques to ameliorate acquiescence bias when scaling indices. These solutions generally field multiple items to construct within-subject corrections. Methods applied include Item Response Theory models, factor analysis, or simple summing across within-subject randomization. For a review, see Billiet and McClendon (Reference Billiet and McClendon2000).

Unfortunately, psychometric methods for scaling indices can be less useful for social scientists interested in beliefs about specific items. Of the 28 studies we mentioned above that surveyed respondents about conspiratorial beliefs, only 3 created indices of belief. Most studies were interested in particular beliefs or specific sets of questions. While some studies are careful to include both true and false statements in the survey, no studies used within-question randomization by negating the statement, the approach we propose below. Furthermore, most of the studies subsequently conducted separate analyses of true and false statements, making it difficult to balance acquiescence bias across the two types of statements.

Social scientists often ask questions about political beliefs giving respondents agreeable response options (“True,” “Yes,” “Agree,” “Likely so,” etc.). If acquiescence arises as a heuristic response when subjects do not know how to answer, bias could be prevalent when (a) asking about topics the subject has not before encountered such as political rumors or conspiracies, or (b) eliciting beliefs about complicated matters of fact beyond the subject’s personal experience (e.g., Schaffner Reference Schaffner2020).Footnote ⁴ The consequences of bias increase in the rarity of the target belief. The smaller the fraction of the population that holds a belief—e.g., when asking about outlandish conspiracies—the more likely agreeable responses are due to acquiescence bias rather than actual belief.

For example, what should we conclude if we found that 30% of subjects respond “True” to the question “True or False: Changes to the health care system enacted by Congress and the Obama administration created ‘death panels’ which have the authority to determine whether or not a gravely ill or injured person should receive health care?” It might be that 30% did not know how to respond to the question and so used the agreeable response “True” as a default, that 30% truly believe that the Obama administration created death panels, or that it is one of the many combinations of the two types that sum to 30%. Acquiescence bias poses a fundamental threat to inference about political conspiracies and factual beliefs about politics.

2 Measuring Acquiescence Bias in Studies of Conspiracies, Rumors, and Facts

In this section, we provide a broad overview of our data and initial results before presenting details on different elicitation methods, studies, and sample populations.

To understand the extent of acquiescence bias in studies of conspiracies, rumors, and facts, we fielded six surveys in China and the United States eliciting subject beliefs about political conspiracies and politically relevant facts. Details of all surveys are in Section B of the Supplementary Material.

We asked three types of questions: (1) questions fielded by other scholars on conspiracies and beliefs in rumors, (2) questions fielded by other scholars on beliefs in political facts, and (3) questions of interest to our own substantive research agendas on factual beliefs and political learning in which we elicited probabilistic beliefs (e.g., Hill Reference Hill2017). In total, we asked 53 questions across the six surveys. All surveys included at least one attention check, and data were not collected and/or not analyzed for respondents who failed these checks.Footnote ⁵ We present results on conspiratorial beliefs in the main text and on misperceptions and political facts in the Supplementary Material.

We fielded two versions of each question. First, we fielded the version as written by the original study, we call this version “positive-keyed.” Second, we fielded an alternative version that was the negative of the original question so that the agreeable response had the opposite meaning (“negative-keyed”). For example, Allcott and Gentzkow (Reference Allcott and Gentzkow2017) asked a nationally representative sample after the 2016 election “At the time of the 2016 election, would your best guess have been that this statement was true?”:

The response options were “Yes, true,” “Not sure,” or “No, false.” We randomized whether our respondents received the original version or the alternative version:

It is important to note that positive-keyed does not mean the content of the statement or conspiracy theory is positive, but rather that the agreeable response option aligns with the original version of the question. In other words, some positive-keyed questions include a “NOT” phrase, and some negative-keyed questions do not. For example, the positive-keyed question of the conspiracy theory surrounding Barack Obama’s birth certificate is phrased, “President Barack Obama was not really born in the United States and does not have an authentic Hawaiian birth certificate,” because this was the original version of the question; whereas the negative-keyed version is phrased, “President Barack Obama was born in the United States and has an authentic Hawaiian birth certificate.”

Randomization occurred at the subject-question level, so each subject received a mix of positive-keyed and negative-keyed questions. Creating a negated version of each item was not always trivial. In the Supplementary Material, we limit analysis to unambiguous negations without consequence to substantive conclusions. We present question wordings in Tables A1–A4 in the Supplementary Material.

For each question with an Agree/Disagree, True/False, or Yes/No response, we calculated the proportion of subjects agreeing with the positive-keyed statement—the original version of the conspiracy theory, rumor, or fact. For ease of comparison, we recode the negated question so that responses have the same target belief as the original, an endorsement of the statement presented in the positive-keyed version. We then code “don’t know” and “not sure” responses as not agreeing. To make this concrete, suppose that a statement had three response options: True, False, and Don’t Know. We code as agreement either answering True to the positive-keyed version or False to the negative-keyed version. In each case, Don’t Know is pooled with disagreement.Footnote ⁶

Absent acquiescence bias, the two percentages should be equivalent, subject to sampling variability. Acquiescence bias, in contrast, would push responses to the two versions in opposite directions with greater estimated endorsement for the positive-keyed wording.

The American National Election Studies (ANES) in recent years, to its credit, has fielded an instrument to elicit beliefs on misinformation with a feature to counteract acquiescence bias. This instrument is similar in spirit to our approach. In a first question, the instrument asks the respondent to indicate which of the two versions, roughly positive- and negative-keyed, they believe more likely true. In a second question, the instrument asks how confident they are in that position.Footnote ⁷ We discuss the benefits to this approach in Section D of the Supplementary Material and conclude that the instrument is likely to more effectively mitigate acquiescence bias than the instruments commonly fielded by other researchers. We also show, however, that scholars might want to use our methods with the ANES instrument to mitigate acquiescence bias in the second confidence question.

In Figure 1, we present the results of the 53 survey questions we fielded. On the x-axis, we plot the estimated agreement with the conspiracy theory, rumor, or fact when the question is positive-keyed. On the y-axis, we plot estimated agreement with the conspiracy theory, rumor, or fact when the question is negative-keyed, the negated form of the original version. Note that the responses of the negative-keyed version have been recoded, so they exist on the same scale as the positive-keyed and, therefore, absent bias the points in this plot should fall approximately on the 45 degree line.

Figure 1 Effect of question wording on agreement with rumors and facts. “*” indicates positive and negative-keyed estimates statistically distinct at $p<.05$ two-tailed.

All but one point falls below the 45 degree line, meaning that the positive-keyed version of the question has a larger estimate of endorsement than the negative-keyed version.Footnote ⁸ In some questions, the points are quite close to the line; for example, Question 21 fielded in the United States on whether the Chinese government created the coronavirus as a biological weapon does not exhibit large amounts of acquiescence bias. In other questions, the positive-keyed version of the question produces a drastically larger estimate of belief. For example, Question 26 on whether or not the FBI director alerted Congress about new emails on Hillary Clinton’s server on October 28, 2016 yielded an agreement of 64% when asked in the positive-keyed version, but only 22% when asked in a negative-keyed version.

3 Robustness of Main Findings to Question Negation and Country of Sample

Not all original questions yielded a simple alternative wording of unambiguous logical equivalence. Skeptical readers might wonder if the divergence in rates of endorsement between positive- and negative-keyed versions is due to logical non-equivalence rather than acquiescence bias. To evaluate this possibility, we reproduce Figure 1 in Section F of the Supplementary Material for 35 questions that we identified of unambiguous logical equivalence. The alternative version of these questions simply adds or subtracts the word “not” along with minor modification of verb tenses. Figure A4 in the Supplementary Material reproduces the pattern of Figure 1 with all items below the 45 degree line and many far below. We find that, if anything, ambiguous alternative question wordings lead to less acquiescence bias with mean magnitudes of 8.9 (unambiguous) and 7.3 (ambiguous). We also find that our results hold separately for U.S. and China samples (Table A6 in the Supplementary Material).

4 Correlates of Conspiracy Theory Beliefs and Acquiescence Bias

In addition to the incidence of conspiratorial beliefs and misperceptions in the population, correspondence between such beliefs and citizen characteristics interests scholars. For example, Allcott and Gentzkow (Reference Allcott and Gentzkow2017) report that having beliefs aligned with the partisan or ideological implications of a news headline corresponds to greater endorsement of the veracity of that headline. We replicate the Allcott and Gentzkow (Reference Allcott and Gentzkow2017) finding first with responses from our subjects assigned the original version of the question. We then compare conclusions with analysis of negative-keyed questions.

Table 1 presents ordinary least-squares (OLS) regression results of the relationship between subject ideological alignment with the news headline (as coded by the original authors) and endorsement of the news headline, for both positive-keyed and negative-keyed versions of the questions we repeated from Allcott and Gentzkow (Reference Allcott and Gentzkow2017) in our U.S. 2020 survey. As before, we recode responses to negative-keyed versions so that “agreement” indicates agreement with the original wording of the question. Following Allcott and Gentzkow (Reference Allcott and Gentzkow2017), we code agreement with the original wording as 1, “Don’t know” as .5, and disagreement as 0. Standard errors are clustered on the respondent because each answers more than one question.

Table 1 Analysis of responses to questions we repeated from Allcott and Gentzkow (Reference Allcott and Gentzkow2017) in our U.S. 2020 survey. Correlation between aligned ideology and belief in news headlines, positive-keyed versus negative-keyed items. Left two regression show results for all news headlines, and right two regression show results only for items labeled by Allcott and Gentzkow (Reference Allcott and Gentzkow2017) as “Big Fake.” Standard errors clustered on the respondent in parentheses.

The negative-keyed questions (columns 2 and 4) suggest substantively smaller correlation between ideological alignment and reported belief. The magnitude of decline is especially large among the subset of news headline described by Allcott and Gentzkow (Reference Allcott and Gentzkow2017) as “Big Fake,” fake news stories that were mentioned in at least three mainstream media articles. Among these stories, the relationship to ideological alignment is 45% smaller with the negative-keyed question.

The cause of the divergent correlations in Table 1 is that acquiescence bias varies substantially between groups. We evaluate how bias varies by four characteristics of the respondent drawn from existing work on acquiescence bias and misperceptions. First, we consider how it varies with political ideology. Second, we consider the relationship with age and education, two factors that the older survey methodology literature on acquiescence bias suggests. Third, we consider variation with respondent aptitude at numerical thinking following the measurement instrument of Frederick (Reference Frederick2005) and arguments about the relationship between reasoning and accepting fake news made by Pennycook and Rand (Reference Pennycook and Rand2019) and Stanley et al. (Reference Stanley, Barr, Peters and Seli2021).

In Table 2, we present regressions pooling positive- and negative-keyed questions and interacting an indicator of positive-keyed with ideological alignment and with characteristics of the subject. We find substantively large interactions with ideological alignment. Questions aligned with respondent ideology exhibit more acquiescence bias, especially for “Big Fake” statements.

Table 2 Analysis of responses to questions we repeated from Allcott and Gentzkow (Reference Allcott and Gentzkow2017) in our U.S. 2020 survey. Impact of Allcott and Gentzkow (Reference Allcott and Gentzkow2017) question wording on belief varies by ideological alignment. Left regression is all statements, and right regression is “Big Fake” statements. Standard errors clustered on the respondent in parentheses.

In Tables A7 and A8 in the Supplementary Material, we evaluate how acquiescence bias affects correlations for questions from Oliver and Wood (Reference Oliver and Wood2014) and Jamieson and Albarracin (Reference Jamieson and Albarracin2020). We find similar results. The magnitudes of correlation between endorsement of conspiracies and covariates and between misperception of facts and covariates depend on whether the question was fielded as original or negated.

In sum, for each of the studies that we refielded, we find that magnitude of acquiescence bias varies by characteristics of the respondent. Acquiescence bias for these questions is more common among very conservative subjects, very liberal subjects, younger subjects, those with innumeracy, and those with lower education.

5 Accounting for Acquiescence Bias

Fielding positive- and negative-worded versions of 53 questions eliciting beliefs shows that acquiescence bias influences survey estimates of three quantities of interest to political scientists: rates of endorsement of conspiracies, rates of political misperceptions, and correlations between features of individuals and their beliefs. In this section, we present two simple statistical procedures to estimate (a) population beliefs controlling for acquiescence bias and (b) correlations purged of acquiescence bias.

Our methods correct sample data to provide an acquiescence-free estimate of population quantities. We show how random assignment of positive- and negative-keyed versions of each item allows for estimation of the magnitude of and a correction for bias.

Consider a conspiratorial or factual statement about the world, for example, that Barack Obama was born in the United States. Let the random variable $Y_{i}$ represent the response by subject i to a measurement instrument—e.g., a survey question—used to measure subject i’s belief about the statement. Let $Y^{*}_{i}$ be the subject’s belief about the statement could it be elicited without acquiescence bias.

Our replication studies strongly suggest that the response $Y_{i}$ in many cases is perturbed by acquiescence bias such that $E(Y_{i}) \neq Y^{*}_{i}$ . To characterize the bias $E(Y_{i} - Y^{*}_{i})$ , define $Y_{i}(p)$ and $Y_{i}(n)$ as the responses of subject i were she asked the positive-keyed (p) or negative-keyed (n) version of the measurement instrument. $Y_{i}(p)$ and $Y_{i}(n)$ should be coded so that increasing values indicate increasing endorsement of the belief, not increasing agreement with the question asked. For example, if question version p is “True or False: Barack Obama was born in the United States” and version n is “True or False: Barack Obama was not born in the United States,” and if $Y_{i}(p)=1$ represents response “True” and $Y_{i}(p)=0$ represents the response “False,” then $Y_{i}(n)=1$ would represent “False” and $Y_{i}(n)=0$ would represent “True.” For both versions, the original response “True” is the agreeable response possibly subject to acquiescence bias.

The data-generating process of subject i measured response $Y_{i}$ is

(1) $$ \begin{align} \nonumber & Y_{i}(p) = Y^{*}_{i} + \delta_{i}, \\ & Y_{i}(n) = Y^{*}_{i} - \delta_{i}\\[-24pt]\notag \end{align} $$

with $\delta _{i}$ acquiescence bias for subject i.

Let $D_{i}$ represent the version of instrument fielded such that $D_{i}=1$ if i responds to the positive-key version p and $D_{i}=-1$ if i responds to the negative-key version n. We then consolidate (1) into

(2) $$ \begin{align} Y_{i}(D_{i}) = Y^{*}_{i} + D_{i}\delta_{i}.\\[-24pt]\notag \end{align} $$

The observed response $Y_{i}(D_{i})$ is the acquiescence-free belief $Y^{*}_{i}$ plus (when $D_{i}=1$ ) or minus (when $D_{i}=-1$ ) the subject’s acquiescence-bias $\delta _{i}$ . Given data-generating Equation (2), consider an OLS regression of Y on D with a representative sample from the target population

$$ \begin{align*} Y_{i} = \alpha + \beta D_{i} + \varepsilon_{i}\\[-24pt] \end{align*} $$

with $\varepsilon _{i}$ an independent and identically distributed error term. When the expected value of $\varepsilon _{i}$ is zero, the coefficient $\alpha $ estimates the population average belief $E(Y^{*}_{i})$ purged of acquiescence bias. The coefficient $\beta $ estimates the population average acquiescence bias $\Delta = E(\delta _{i})$ under the usual assumptions for ordinary least-squares regression. This regression might also use post-stratification survey weights to target a population-level average in the presence of individual heterogeneity (assuming random assignment of D).

5.1 Correlates

In addition to population rates, our method can be extended to mitigate bias in correlations. To model the relationship between beliefs and characteristics of the subject, substitute the quantity $Y^{*}_{i}$ in Equation (2) with the linear combination $x_{i}'\gamma $ , x a k-vector of covariates and $\gamma $ a k-vector of coefficients. Thus, $\gamma $ represents the expected correspondence between individual characteristics $x_{i}$ and acquiescence-free beliefs $Y^{*}_{i}$ . This substitution leads to the updated data-generating process

(3) $$ \begin{align} Y_{i}(D_{i}) = x_{i}'\gamma + D_{i}\delta_{i}. \end{align} $$

To see the problem of acquiescence bias for estimating correlates of political beliefs, consider a sample of size $\mathcal {N}$ . Define the $\mathcal {N}$ -vectors Y, D, and $\delta $ for the sample values of $Y_{i}$ , $D_{i}$ , and $\delta _{i}$ . Define X the $\mathcal {N}$ -by-k matrix of covariates. The OLS estimate of $\gamma $ is

$$ \begin{align*} \hat{\gamma} = (X'X)^{-1}X'Y.\\[-24pt] \end{align*} $$

Substituting the known data-generating process of Y, we have

(4) $$ \begin{align} \nonumber \hat{\gamma} & = (X'X)^{-1}X'(X\gamma + D\delta) \\ & = \gamma + (X'X)^{-1}X'D\delta.\\[-24pt]\notag \end{align} $$

The OLS estimator of the k correlations $\gamma $ is biased by the second term in (4), the covariance between $x_{i}$ and $\delta _{i}$ . Covariance between $x_{i}$ and $\delta _{i}$ was exactly the problem diagnosed by the The American Voter discussed in Section 1; because studies had fielded only positive-keyed versions of questions measuring authoritarianism, correlation between acquiescence bias and education inflated estimates of the relationship between authoritarianism and education.

Equation (4) suggests two options to ameliorate bias in $\hat {\gamma }$ . In each, the analyst assigns at-random positive- and negative-keyed versions of each item. In the first solution, regress $Y_{i}$ on covariates of interest $x_{i}$ , a variable $D_{i}$ taking the values 1 and $-1$ indicating version positive and negative, and an interaction between $D_{i}$ and $x_{i}$ :

$$ \begin{align*} Y_{i} = \beta_{0} + \beta_{1}x_{i} + \beta_{2}D_{i} + \beta_{3}D_{i}x_{i} + \epsilon_{i}.\\[-24pt] \end{align*} $$

The coefficient(s) on $x_{i}$ , $\beta _{1}$ , estimates the correlation between $x_{i}$ and $Y^{*}_{i}$ free of acquiescence bias. The coefficient $\beta _{2}$ is an estimate of average acquiescence bias, and the coefficient(s) $\beta _{3}$ estimate heterogeneity of acquiescence bias related to $x_{i}$ . As before, one could estimate this regression with post-stratification survey weights.

Second, the analyst can implement weighted least-squares (WLS) without including the variable $D_{i}$ in the regression. When the vector D is perfectly balanced—when the number of positive- and negative-keyed responses is exactly equal—its expected value is zero because one $D_{i}=1$ offsets each $D_{i}=-1$ . When the vector D is assigned at random, the expected covariance of $D_{i}$ and $\delta _{i}$ is zero. When the covariance of $D_{i}$ and $\delta _{i}$ is zero, the expected values of their product, $D_{i}\delta _{i}$ , is zero. Therefore, the expected value of the bias terms in (4) is zero when D is balanced and assigned at random.

To achieve exact balance on D, define the matrix W with elements on the diagonal $\frac {\mathcal {N}}{2N}$ for the N subjects asked question version p and $\frac {\mathcal {N}}{2\tilde {N}}$ for the $\tilde {N}$ subjects asked question version n, off-diagonal cells zero, $N + \tilde {N} = \mathcal {N}$ . Then

(5) $$ \begin{align} \hat{\gamma}_{WLS} = (X'WX)^{-1}X'WY\\[-24pt]\notag \end{align} $$

is an unbiased estimate of $\gamma $ . If the dataset includes post-stratification weights, the values N and $\tilde {N}$ should be the sum of the weights in each assignment rather than the count of observations, and the diagonals of the weight matrix W should be the product of the terms above and the original survey weight. With survey weights, the regression method is likely to be easier to implement than weighting.Footnote ¹²

We emphasize that these fixes depend on the untested assumption that acquiescence bias does not vary by key. Because of this assumption, we recommend that researchers present analysis separately by version of the question in addition to implementing the proposed solutions.

6 Applying the Method

We apply the WLS approach to examine the correlation between ideology and belief in conspiracy theory in each of the studies. For each question, we calculated the weights for positive-keyed and negative-keyed responses as $\frac {\mathcal {N}}{2N}$ and $\frac {\mathcal {N}}{2\tilde {N}}$ . Because of the large number of items, we present coefficient plots instead of regression tables. While we plot and discuss the correlation between conspiratorial belief and the subject identifying as very conservative in the main body, we apply the method to coefficients for very liberal, numeracy, age, and education and present full regression tables in Section I of the Supplementary Material.

In Figure 2, we plot coefficients, 84% confidence intervals (thick line), and 95% confidence intervals (thin line) on the very conservative indicator for each question for (1) the sample assigned the positive-keyed version of the question, (2) the sample assigned the negative-keyed version of the question, and (3) the WLS method combining the two (point with solid line). To be consistent with the original coding in Allcott and Gentzkow (Reference Allcott and Gentzkow2017), we coded agreement with the original version of the question as 1, don’t know as .5, and 0 as disagreement, although coding don’t know as zero produces very similar results. We shade the background for coefficients where we would reject the null hypothesis of no relationship between very conservative and agreement with one question wording but not reject with another. For half of the questions, the result of this statistical test depends on question wording.

Figure 2 Coefficient indicating the correlation between very conservative and belief in the conspiracy theory. Questions refielded from Allcott and Gentzkow (Reference Allcott and Gentzkow2017), Oliver and Wood (Reference Oliver and Wood2014), and Jamieson and Albarracin (Reference Jamieson and Albarracin2020). The thick and thin lines extend to 84% and 95% confidence intervals.

7 Recommendations and Conclusion

We believe that our evidence suggests that acquiescence bias causes substantively important problems for research on political conspiracies and beliefs. We hope to have brought this challenge to the attention of scholars and to have provided knowledge and methods to improve future research. We see as particularly promising the possibility that acquiescence bias might be related to findings on framing effects (e.g., Druckman Reference Druckman2001) if different frames might be keyed in positive versus negative directions. We would also be interested to see new work on the potential influence of acquiescence bias on survey measurement of citizen policy views.

While we do not believe there is a one-size-fits-all recommendation because each topic of interest will have its own idiosyncracies, our investigation brings forward a set of suggested practices.

First, we recommend fielding both positive- and negative-keyed instruments to elicit beliefs. Examining differences in estimates and presenting them to readers should be the first practice. We would then, in general, recommend combining results from the two instruments with one of our proposed estimators.

Second, we would encourage scholars to consider multiple-choice or other instruments without agreeable responses. This approach, however, presents the first-order challenge of selecting the appropriate set of response options. In Section G of the Supplementary Material, we analyze an alternative multiple-choice approach we fielded in our surveys similar to the suggestion in Clifford, Kim, and Sullivan (Reference Clifford, Kim and Sullivan2019). We do not find evidence that the approach is superior but would advocate future research.

The third option is to field an instrument similar to recent American National Election Studies misinformation items (see Section D of the Supplementary Material). We would advocate future research to determine the trade-offs between the ANES two-question approach and our single-question approach. We show in the Supplementary Material that both approaches can benefit from our WLS statistical fix. One challenge for future work is that there is no absolute benchmark with which to evaluate different instruments.

We close with a more general recommendation. Measurement challenges should be of central concern for scholars interested in sensitive or complicated political beliefs. Best practice in survey research is to pilot, test, and evaluate different measurement instruments. Too often scholars put one question on a survey without testing and validating and then assume the responses to that particular item accurately reflect the beliefs of their subjects. We encourage scholars to consider how the choice of instrument might materially influence the inferences they draw about political conspiracies and beliefs.

Funding and Research with Human Subjects

Data collected for the 2017–2018 U.S. study by Time-Sharing Experiments for the Social Sciences, NSF Grant 0818839, Jeremy Freese and James Druckman, Principal Investigators. We thank in part UC San Diego (UCSD), Academic Senate Research Award for support. In addition, for support for some of our surveys, we thank the “China from the Ground Up” project under the auspices of the China Data Lab at UCSD, supported by the Carnegie Corporation of New York, Henry Luce Foundation, and private donors to the 21st Century China Center. The experiments presented here are approved by the UCSD Human Research Protections Program.