Moderator Placement in Survey Experiments: Racial Resentment and the “Welfare” versus “Assistance to the Poor” Question Wording Experiment

Abstract Researchers face difficult decisions about whether to ask potential moderators before or after a survey experiment. Competing concerns exist about priming respondents before the experiment and about introducing post-treatment bias. We replicate the classic “welfare” versus “assistance to the poor” survey experiment, randomly assigning respondents to be asked a battery of racial resentment questions either before or after the question wording experiment. We find little evidence that the question wording effect is different between those who are asked about racial resentment before versus after the experiment. Furthermore, we find little evidence that measured racial resentment is affected by this ordering or by the question wording treatment itself.

"Welfare" versus "assistance to the poor" and spending views In a now classic survey experiment, Smith (1987) and Rasinski (1989) show that when a policy is described as "welfare" as opposed to "assistance to the poor," respondents are more likely to say there is too much spending on the policy. 2e analyze a new survey that allows us to replicate these basic results while also testing important hypotheses about the impact of question ordering.
We recruited 1,590 participants for a short survey that included the question wording experiment about spending views as well as a standard four-question battery of racial resentment items (Kinder and Sanders 1996). 3Importantly, we randomly assign the order of the question wording experiment and racial resentment battery. 4We recode each item in the racial resentment battery so that 0 (1) is the least (most) racially resentful response and create a racial resentment scale by averaging the four variables together.This racial resentment scale serves as our potential moderator of interest.We pre-registered this study and our analysis plan with the OSF [https://doi.org/10.17605/OSF.IO/JYZBE].
Table 1 presents a basic (naive) analysis of the experiment, ignoring possible issues with moderator order.Model 1 shows that the estimated effect of the "welfare" wording on spending views (1 = "too much"; −1 = "too little"; 0 = "about the right amount") is .27and highly significant.We might also suspect that racial resentment predicts respondents' views on welfare spending (Federico 2004) and Model 2 confirms this.5 Furthermore, Model 3 shows that the effect of the welfare wording treatment is much stronger for respondents with higher levels of racial resentment.
But Table 1's results are a prime example of the thorny issues discussed above.Because Models 2 and 3 condition on racial resentment, we may be wary of measuring racial resentment post-treatment.But if the battery of racial resentment questions is asked before the main spending question, we may worry that respondents are primed by being asked questions about race and the ability of different groups to get ahead in society.Because our study randomized the order of these questions, we are able to directly estimate these question order effects.

Does asking about racial resentment change the question wording effect?
In Table 2, Model 1 presents the results of a linear regression predicting spending views with an indicator for the "welfare" wording, RRpost (an indicator for whether racial resentment was measured after, as opposed to before, the question wording experiment), and an interaction between these two variables.We see that the welfare wording has a strong effect on spending views, as in Table 1.
2 See also Huber and Paris (2013) for an argument that respondents perceive the policies behind these two terms differently.The survey was fielded to MTurk participants using the CloudResearch platform.The online appendix contains more information about how the survey was fielded as well as basic descriptive statistics for the sample and full question wordings.
But there is little evidence that the ordering of the racial resentment battery affects responses to the spending question.More importantly, the coefficient on the interaction between welfare wording and being asked racial resentment items post-treatment is small in magnitude and highly insignificant (p = .59),suggesting that asking the racial resentment items before the question wording experiment does not meaningfully change the experiment's treatment effect.The point estimate for this coefficient is -.04, which, taken at face value, would imply that the effect of the "welfare" wording is roughly 15% smaller for those who are asked the racial resentment battery post-treatment, but it should be noted that our study is not adequately powered to detect such a small effect difference if it were real.
Although not our main focus, Model 2 shows that more resentful respondents are more likely to think too much is spent and racial resentment has a larger impact under the "welfare" wording treatment. 6There is suggestive evidence that racial resentment has a smaller association when asked post-treatment, while the triple interaction term is not significant, although it is estimated with a decent amount of uncertainty.
Is racial resentment affected by the question wording treatment?Another way to explore the possibility of post-treatment bias from conditioning on moderators measured after an experiment is to ask whether the average measured level of the moderator itself appears to be affected by the treatment. 7Table 3 shows the results of a linear regression predicting racial resentment with RRpost and an interaction between RRpost and the "welfare" question wording. 8Both coefficient  A related question, which we do not directly address here, is whether measuring moderators after treatment affects the accuracy of the moderator as a measure of the intended concept.This more subtle issue is not directly addressed here.8 Note we have omitted the welfare word predictor by itself and only included it through the interaction with RRpost.The coefficient on welfare word alone would represent the effect of the "welfare" question wording on racial resentment for respondents who were asked the racial resentment battery before the question wording experiment.Thus, our specification assumes the question wording randomization cannot affect racial resentment if the racial resentment items are asked before the spending question wording is shown to a respondent.estimates are small in magnitude and highly insignificant.It should be noted that the estimated difference in racial resentment between those receiving the racial resentment battery first and those receiving it afterwards who were shown the "welfare" wording might be termed marginally significant (p = .12),but this estimated difference of -.03 is small in magnitude.To put this in perspective, the point estimates for these coefficients would imply that compared to respondents who were asked the racial resentment battery at the start of the survey, those asked the racial resentment items post-treatment would have racial resentment levels on average 2% lower in the "assistance to the poor" condition, and just over 7% lower in the "welfare" condition.
Collectively, these results provide little evidence that measured racial resentment differs meaningfully on average based on the ordering and wording of the experiment (note that testing the null hypothesis that both slope coefficients are equal to zero produces a p-value of .30).Though we are primarily interested in whether moderator order affects the treatment, our study design also allows us to test the oppositedoes the question wording experiment affect the moderator?We do not find strong evidence that it does.
The appendix also shows the results of Kolmogorov-Smirnov tests comparing the overall distribution of racial resentment between these three groups.None of these tests for equality of distribution between the three groups (presented in detail in Appendix) reached conventional significance levels. 3 7

Table 2
Regression Predicting Spending Preferences on Welfare/Assistance to the Poor

Table 3
Regression Predicting Racial Resentment Linear regression coefficient estimates with standard errors underneath in parentheses.