Self-Reported Political Ideology


 American politics scholarship has relied extensively on self-reported measures of ideology. We evaluate these widely used measures through an original national survey. Descriptively, we show that Americans’ understandings of “liberal” and “conservative” are weakly aligned with conventional definitions of these terms and that such understandings are heterogeneous across social groups, casting doubt on the construct validity and measurement equivalence of ideological self-placements. Experimentally, we randomly assign one of three measures of ideology to each respondent: (1) the standard ANES question, (2) a version that adds definitions of “liberal” and “conservative,” and (3) a version that keeps these definitions but removes ideological labels from the question. We find that the third measure, which helps to isolate symbolic ideology from operational ideology, shifts self-reported ideology in important ways: Democrats become more conservative, and Republicans more liberal. These findings offer first-cut experimental evidence on the limitations of self-reported ideology as a measure of operational ideology, and contribute to ongoing debates about the use of ideological self-placements in American politics.


| S2 A Sample Characteristics
Table S1 shows that our sample characteristics are comparable to the national benchmarks.

B Balance across Experimental Groups
We use equivalence testing recommended by Hartman and Hidalgo (2018) to test the covariate balance between experimental groups.In an equivalence test, difference between group means is the null hypothesis, and equivalence is the alternative hypothesis. 1 Figures S1-S3 show that, under the default equivalence ranges (i.e., ±0.36σ , with σ the pooled SD of the tested covariate), all important demographic and attitudinal variables are balanced across the three experimental groups.Our two-sample Kolmogorov-Smirnov tests show the following: • The control group vs. the Add Definitions condition: D = 0.0606, p = 0.0212.There is a statistically significant difference between the distributions of self-reported ideology in the control group and the Add Definitions condition.Defining the terms "liberal" and "conservative" for respondents changes the distribution of self-reported ideology in the full sample.
• The control group vs. the Subtract Labels condition: D = 0.0432, p = 0.1917.There is a statistically insignificant difference between the distributions of self-reported ideology in the control group and the Subtract Labels condition.Our main text shows that this masks important heterogeneity (e.g., Table 2, Figure 5, Figure 6).
• The Add Definitions condition vs. the Subtract Labels condition: D = 0.0808, p = 0.0046.
There is a statistically significant difference between the distributions of self-reported ideology in the Add Definitions condition and the Subtract Labels condition.Taking away the ideological labels when defining them changes the distribution of self-reported ideology in the full sample.

| S7
C.2 Tests for Equality of Correlations between Self-Reported Ideology and

Fiscal Preferences
In our pre-analysis plan, we preregistered on page 10 that we would adopt the bootstrap approach advocated by Rousselet, Pernet, and Wilcox (2021) to compare (1) the Pearson's r between fiscal preferences and self-reported ideology in the control group and (2) the Pearson's r between fiscal preferences and self-reported ideology in the Add Definitions condition or the Subtract Labels condition.
The question wording for fiscal preferences is taken directly from ANES: "Some people think the government should provide fewer services in order to reduce spending.Suppose these people are on one end of the scale, at point 1.Other people feel that it is important for the government to provide many more services even if it means an increase in spending.Suppose these people are at the other end, at point 7. And, of course, some other people have opinions somewhere in between.
• The difference in correlations between self-reported ideology and fiscal preferences in the control group and the Add Definitions condition is 0.049 (p = 0.221).
• The difference in correlations between self-reported ideology and fiscal preferences in the control group and the Subtract Labels condition is −0.058 (p = 0.218).
• The difference in correlations between self-reported ideology and fiscal preferences in the Add Definitions condition and the Subtract Labels condition is −0.107 (p = 0.035). 2 2. While we did not preregister this analysis, we report it for completeness.

| S8 C.3 Tests for Equality of Correlations between Self-Reported Ideology and Preferences for Government Responsibility
In our pre-analysis plan, we preregistered on page 10 that we would adopt the bootstrap approach advocated by Rousselet, Pernet, and Wilcox (2021) to compare (1) the Pearson's r between preferences for government responsibility and self-reported ideology in the control group and (2) the Pearson's r between preferences for government responsibility and self-reported ideology in the Add Definitions condition or the Subtract Labels condition.
The question wording for preferences for government responsibility is taken directly from ANES: "Some people feel the government should see to it that every person has a job and a good standard of living.Suppose these people are at one end of a scale, at point 1.Others think the government should just let each person get ahead on their own.Suppose these people are at the other end, at point 7. And, of course, some other people have opinions somewhere in between, at points 2, 3, 4, 5, or 6.Where would you place yourself on this scale?"As preregistered, this variable ranges from 1 ("government should see to jobs and standard of living") to 7 ("government should let each person get ahead on own").
• The difference in correlations between self-reported ideology and preferences for government responsibility in the control group and the Add Definitions condition is 0.089 (p = 0.035).
• The difference in correlations between self-reported ideology and preferences for government responsibility in the control group and the Subtract Labels condition is 0.194 (p < 0.001).
• The difference in correlations between self-reported ideology and preferences for govern- ment responsibility in the Add Definitions condition and the Subtract Labels condition is 0.105 (p = 0.022). 3 Thus, our label-free measure of ideology (i.e., the measure adopted by the Subtract Labels condition) appears to perform much better in capturing respondents' preferences for government responsibility than the standard ANES measure does.

C.4 Heterogeneity by Education
In our pre-analysis plan, we preregistered on page 11 that we would conduct subgroup analysis by education.Figure S4 shows how ideological knowledge differs between college and non-college graduates.The average number of correct responses to the ideology questions was 2.41 among college graduates and was 1.55 among non-college graduates (β = 0.86, 95% CI = [0.76,0.95], n = 2, 774).Thus, highly educated individuals appear to be more familiar and conversant with ideological terms.We explore the existence of heterogeneous treatment effects by education.To do so, we estimate the following equation: Self-Reported Ideology 3. While we did not preregister this analysis, we report it for completeness.

| S10
Graduate i + ε i , where Self-Reported Ideology i is respondent i's self-reported ideology in the corresponding treatment condition, Add Definitions i and Subtract Labels i indicate i's treatment conditions, and College Graduate i is a dummy variable indicating whether i already graduated from college.Estimates of γ 1 and γ 2 are 0.16 (SE = 0.14) and -0.05 (SE = 0.14) respectively, suggesting a lack of treatment effect heterogeneity.

C.5 Word Counting for Open-Ended Responses
In our pre-analysis plan, we preregistered on page 12 that we would count how many self-reported liberals, in an open-ended question, 4 mention any of the following terms: "Democrat," "Democrats," and "Democratic Party."We also preregistered that we would count how many self-reported conservatives, in another open-ended question, 5 mention any of the following terms: "Republican," "Republicans," and "Republican Party."We find 17 respondents for the former and 22 for the latter.To provide a reference point, we also counted how many respondents mentioned any of the following terms: "activ*", "chang*", "tradition*", "interven*" (where the asterisks indicate any possible combinations of letters).Although this dictionary for word counting is not preregistered, they are the key terms that appear in Lowi et al.'s (2019) definitions of "liberal" and "conservative." We find that 36 self-reported liberals and 40 self-reported conservatives mentioned these terms.This appendix documents and lists any deviations of our paper from the pre-analysis plan.The pre-analysis plan is available at https://osf.io/87bvs.

D.1 Sample
• Recruitment: As preregistered, we recruited our respondents via Lucid.
• Sampling: Consistent with our pre-analysis plan, we set demographic benchmarks for Lucid based on sex, age, race, and household income.Our benchmarks in sex and age are drawn from the 2019 American Community Survey (Table S0101).For race, we draw from the 2020 Decennial Census (PL 94-171, Tables P3-P4).For household income, we draw from the 2021 CPS Annual Social and Economic Supplement (Table HINC-01).Table S1 shows that our sample is close to these preregistered benchmarks.
• Size: As stated in the pre-analysis plan, the targeted sample size was 2,500 American adults.
An increase in research budget allowed us to recruit more respondents.This increased our study's power, especially in terms of detecting treatment effect heterogeneity.
• Inattentiveness: As preregistered, we excluded respondents whose survey completion time is less than 5 minutes.This procedure dropped 529 respondents, accounting for 12.14% of the original sample.

D.2 Measurement
Only the measurement of ideological knowledge deviates from our preregistration.We explain that in detail in Appendix E. All other variables reported in this study, including the demographics and outcome variables, are measured and coded in accordance with the pre-analysis plan.

D.3 Analysis
Our reported analyses in the main text closely adhere to our pre-analysis plan.For transparency, we record which reported analyses are preregistered, and which are not, in Table S2.
Self | S13 E Ideological Understanding before Dropping the Free Trade Item In our survey, we asked a fifth question for the ideological knowledge test that concerns free trade.
The question item was: "In general, free trade is good for our country."We defined the correct answer as "Neither."The percentage of correct responses was only 19.91%.This unusually low accuracy raises concerns about measurement error.To test its impact on the internal consistency of our measure of ideological knowledge, we find that the Cronbach's alpha of the variable ideological knowledge drops from α = 0.65 to α = 0.51 if we include the free trade item.Thus, we deviate from our pre-analysis plan by analyzing this outcome variable with the free trade item dropped.
For completeness, however, we also present a version of analysis of ideological understanding with the free trade item included.Keeping the free trade item, our ideological knowledge variable now takes a 6-point scale and ranges from 0 ("Least knowledgeable") to 5 ("Most knowledgeable").Our reanalysis of ideological understanding, as reported below, suggests that the substantive results remain unchanged.
Figure S5 shows the distribution of the new ideological knowledge variable.Figure S6 shows the distribution of the new ideological knowledge variable across demographic subgroups.Note: Conservatives are respondents who selected "slightly conservative," "conservative," or "extremely conservative" under ANES's 7-point ideology question.Liberals are those who selected "slightly liberal," "liberal," or "extremely liberal."Moderates are those who selected "moderate; middle of the road."Error bars represent 95% confidence intervals.Note: Conservatives are respondents who selected "slightly conservative," "conservative," or "extremely conservative" under ANES's 7-point ideology question.Liberals are those who selected "slightly liberal," "liberal," or "extremely liberal."Error bars represent 95% confidence intervals.We find that black and female respondents were more likely to report "Don't know" than their counterparts (Figure S21).On average, the number of "Don't Know" responses to the ideological knowledge questions among Blacks was 0.15 higher than that among non-Blacks (p = 0.02, n = 2, 774), and the number among female respondents was 0.44 higher than the number among male respondents (p < 0.001, n = 2, 774).

Figure S1 .Figure S2 .
Figure S1.Covariate Balance between the Control Group and the Add Definitions Condition

Figure S3 .
Figure S3.Covariate Balance between the Add Definitions Condition and the Subtract Labels Condition

Figure S4 .
Figure S4.Distribution of Ideological Knowledge among College and Non-College Graduates 4. The question wording is "In a sentence or two, please tell us what defines a LIBERAL?" 5.The question wording is "In a sentence or two, please tell us what defines a CONSERVA-TIVE?" | S11 D Compliance with the Pre-Analysis Plan

Figure S5 .Figure S6 .Figure S8 .
Figure S5.Distribution of Ideological Knowledge before Dropping the Free Trade Item

Figure S9 .
Figure S9.Distribution of Republicans' Self-Reported Ideology across Experimental Conditions

Figure S12 .
Figure S12.Average Number of Correct Responses by Question Type and Self-Reported Ideology (Speeders Included)

Figure S13 .
Figure S13.Distribution of Ideological Knowledge across Demographic Subgroups (Speeders Included)

Figure S14 .
Figure S14.Distribution of Self-Reported Ideology across Experimental Conditions (Speeders Included)

Figure S15 .
Figure S15.Within-Subjects Differences in Ideology by Partisanship in the Subtract Labels Condition (Speeders Included)

Figure S20 .
Figure S20.Distribution of Ideological Knowledge across Demographic Subgroups with Alternative Operationalization the Extent to Which "Don't Know" Responses Bias the Results

Figure S21 .
Figure S21.The Number of "Don't know" Responses across Racial and Gender Groups

Table S2 .
Compliance of Reported Analyses with the Pre-Analysis Plan

Table S3 .
Percentage of Correct Responses and Don't Knows for Each Ideology Question (Speeders Included) Note: 95% confidence intervals are shown in brackets.Figure S11.Percentage of Correct Responses to Each Ideology Question by Self-Reported Ideology (Speeders Included)

Table S4 .
Self-Reported Ideologies in Different Social Groups Are Changed by Question Wording (Speeders Included) Note: Entries are OLS estimates with robust standard errors in parentheses.