Affective polarization and construct validity

In the LISS panel, feelings towards the 13 political parties represented in the Dutch parliament were assessed with the question: ‘How sympathetic do you find the political parties? You can assign each party a score between 0 and 10. 0 means that you find the party very unsympathetic, and 10 means that you find the party very sympathetic’. In the CSES, the wording was ‘After I read the name of a political party, please rate it on a scale from 0 to 10, where 0 means you strongly dislike that party and 10 means that you strongly like that party’. As recommended by Wagner (Reference Wagner2021) when introducing the applied measures of affective polarization, we categorized ‘Don't know’ responses as missing values. We operationalize affective polarization using the spread of like/dislike across the political spectrum, weighted by the relative vote share of each party, using the following equations (Wagner, Reference Wagner2021):

(1)${\overline{like}}_i=\sum _{p=\mathrm{>1}}^P(v_p·lik{e}_{ip}),$

(2)$WSprea{d}_i=\sqrt{{\sum }_{p=\mathrm{>1}}^P{v}_p·{(lik{e}_{ip}-{\overline{like}}_i)}^{\mathrm{>2}}}.$

Here, ${\overline{like}}_i$ represents the weighted mean like/dislike score for individual $i$, calculated by multiplying the like/dislike score for each party ($lik{e}_{ip}$) by its relative vote share ($v_p$). $WSprea{d}_i$ is the weighted spread of like–dislike ratings for individual $i$ across parties $p$, computed as the root mean square deviation of the like/dislike ratings for each party from an individual's weighted mean like/dislike rating, again considering the relative vote share of each party. Affective polarization is low when individuals exhibit uniform feelings towards all parties – whether positive, negative or neutral – and high when they display strong positive feelings towards specific parties, coupled with negative feelings towards others. By including vote shares, this formula prioritizes the influence of larger parties on an individual's affective polarization score, as it is arguably more consequential if the liked/disliked parties are larger competitors rather than smaller fringe parties (Wagner, Reference Wagner2021).

To operationalize construct validity, we compare affective polarization measures based on all political parties within each country ($N_p$) to affective polarization measures based on all possible combinations of 2 to $N_p-\mathrm{>1}$ parties. For example, in the Netherlands, we calculated one full affective polarization score with all 13 parties and brief scores with all combinations of 2–12 parties (8178 scores in total per participant). We excluded participants only when the calculation of the score was infeasible (i.e., number of evaluated parties $\<\mathrm{>2}$), to avoid removing participants who did not rate lesser known parties. We then operationalize construct validity (i.e., the extent to which the full measures – as the current standard – and the brief measures capture the same construct) by correlating the brief affective polarization measures with the full measures derived from evaluations of all parties. We follow the recommendations of Nunnally and Bernstein (Reference Nunnally and Bernstein1994) and Ellis (Reference Ellis2013) and assume that a construct validity of $r=\mathrm{>0.80}$ or higher is acceptable. Adding more items beyond this point typically yields diminishing returns in terms of increasing construct validity and precision of associations with other variables.

Note that we replicate our results with three alternative affective polarization measures, more specifically, the unweighted spread of like/dislike and the weighted and unweighted mean distance of liking from the favorite party – the latter being more consistent with conceptualizations of affective polarization as revolving mainly around a single political ingroup (Wagner, Reference Wagner2021). The results align well. We provide detailed findings in the online Appendices (see Sections 1 and 6 for unweighted spread, Sections 2 and 8 for unweighted mean distance, and Sections 4 and 11 for weighted mean distance).

Political correlates and predictive validity

To evaluate the predictive validity of the brief measures, we calculated bivariate correlations with often studied correlates of affective polarizations (Torcal & Harteveld, Reference Torcal and Harteveld2025; Wagner, Reference Wagner2024). Specifically, we assess the relationship between affective polarization and satisfaction with democracy, confidence in democracy, party identification, partisan identity strength, political interest and turnout intentions in the LISS and associations with satisfaction with democracy, previous turnout, perceived efficacy of voting, political interest, partisan identity strength and political identification in the CSES. Item formulations can be found in Online Appendices 1.3.1 (LISS) and 6.2.1 (CSES). While the selected political correlates are not an exhaustive representation of all constructs that have been associated with affective polarization, nor are they exclusively related to affective polarization, we focus on them due to their frequent use in recent studies on affective polarization (e.g., see Harteveld & Wagner, Reference Harteveld and Wagner2023; Janssen & Turkenburg, Reference Janssen and Turkenburg2024; Wagner, Reference Wagner2021).

When evaluating the predictive validity of brief measures, we compare the associations between brief measures and political correlates with those observed with the full measures. Thereby, we examine whether brief measures yield results comparable to the current standard measure in terms of relationships with theoretically relevant constructs.

The construct validity of brief measures

The upper panel of Figure 1 shows distributions, means and 95% confidence intervals (CIs) of the correlations between brief and full affective polarization measures (LISS, the Netherlands) across different numbers of parties included in the sample for the calculation of the brief measures (see Online Appendix 3.2 for more details). While measures that include only three parties almost always fail to reach the aspired construct validity of $r=\mathrm{>0.80}$ ($M_r=\mathrm{>0.52}$, $S{D}_r=\mathrm{>0.11}$, $rang{e}_r[\mathrm{>0.13};\mathrm{>0.87}]$), brief measures with four parties already include a larger set of correlations above the specified threshold ($M_r=\mathrm{>0.62}$, $S{D}_r=\mathrm{>0.11}$, $rang{e}_r[\mathrm{>0.24};\mathrm{>0.92}]$). For brief measures that include five parties, the mean correlation with the full measure is $M=\mathrm{>0.70}$ ($S{D}_r=\mathrm{>0.10}$, $rang{e}_r[\mathrm{>0.36};\mathrm{>0.95}]$), with many correlations surpassing $r=\mathrm{>0.80}$. Including additional parties raises the construct validity even more consistently over the specified threshold.

Figure 1. Construct validity – brief affective polarization measures (weighted spread).

Note: Distributions and means of correlations between the full and brief affective polarization scores (weighted spread of like/dislike) with 95% CIs in the Netherlands (LISS, panel A), and across 39 countries (CSES, panel B). The red‐dashed line in both panels represents a correlation of $r=\mathrm{>0.80}$, which was defined as the threshold for acceptable construct validity. Panel B is divided based on the highest number of parties included in the full affective polarization measure as per availability in the CSES. Flags indicate which countries are included in which sub‐panel – they are equally distributed in the sub‐panels and do not correspond to specific data points.

The bottom panel of Figure 1 summarizes the construct validity of the brief affective polarization measures for the countries included in the CSES. Party system size always refers to the highest number of parties available in the dataset for each country.Footnote ² While measures that include only three parties seem to produce an extensive set of combinations that reach the aspired construct validity for party systems with five parties ($M_r=\mathrm{>0.67}$, $S{D}_r=\mathrm{>0.26}$, $rang{e}_r[\mathrm{>0.08};\mathrm{>1.00}]$) or six parties ($M_r=\mathrm{>0.61}$, $S{D}_r=\mathrm{>0.26}$, $rang{e}_r[-\mathrm{>0.15};\mathrm{>0.99}]$), combinations with four parties more consistently provide valid measures for systems with seven parties ($M_r=\mathrm{>0.70}$, $S{D}_r=\mathrm{>0.19}$, $rang{e}_r[\mathrm{>0.14};\mathrm{>0.99}]$), eight parties ($M_r=\mathrm{>0.65}$, $S{D}_r=\mathrm{>0.21}$, $rang{e}_r[\mathrm{>0.02};\mathrm{>0.99}]$) or nine parties ($M_r=\mathrm{>0.65}$, $S{D}_r=\mathrm{>0.17}$, $rang{e}_r[\mathrm{>0.11};\mathrm{>0.99}]$). Including more parties in the measures within each system led to an even more consistent crossing of the threshold of $r=\mathrm{>0.80}$.

In both the LISS and the CSES, we found combinations with a specific number of parties that consistently achieve an acceptable construct validity (e.g., when reducing the number of parties by just one or two). However, briefer versions of the scales (e.g., four/five parties in the LISS and three/four in the CSES), despite performing worse on average, still have a notable proportion of combinations that exceed the specified validity threshold. On the flip side, while combinations of larger numbers of parties are more consistent in achieving sufficient construct validity, they are not always a safeguard against measures with low validity, as evidenced by the wide range of correlations within the same scale length. We conclude that while mean correlations between brief affective polarization measures may be lower or higher than the threshold of $r=\mathrm{>0.80}$, individual brief measures of the same length with specific party combinations can vary from being nearly identical to the full measure ($r\>\mathrm{>0.90}$) to exhibiting very low associations. This raises the question: Is it possible to predict which party combinations are likely to maintain an acceptable validity and if so, how can we identify the most effective combinations of parties to maximize construct validity while minimizing scale length?

To answer this question, we examine the composition of combinations of parties that can be selected to likely attain acceptable construct validity focusing on party size and ideological diversity. Those are the two aspects that have been considered by other researchers who have relied on subsets of parties to measure affective polarization (e.g., Comellas & Torcal, Reference Comellas and Torcal2023; Gidron et al., Reference Gidron, Adams and Horne2019). However, again, the approach has never been formally evaluated. To operationalize ideological diversity, we use the latest version of the Chapel Hill Expert Survey (CHESS; Jolly et al., Reference Jolly, Bakker, Hooghe, Marks, Polk, Rovny, Steenbergen and Vachudova2022) from 2019 for the LISS, and comparable expert assessments available in the CSES. Both provide estimates of parties' ideological positions on a scale from 0 (left) to 10 (right)Footnote ³. We calculate mean ideological diversity as the mean pairwise difference between the ideological positions of parties within each combination of parties. Additionally, we operationalize the mean party size for each combination of parties as the average vote share of all parties within a given combination, based on the most recent lower house elections.

To examine the relationship between construct validity and both characteristics in the Netherlands (LISS), we employed a linear regression with the correlation between the brief and full measures of affective polarization as the dependent variable, and mean ideological diversity and mean party size within the combination of parties (both z‐standardized, i.e., $M=\mathrm{>0}$, $SD=\mathrm{>1}$) and their interaction as predictors. We control for the number of parties included in the measure. We find a statistically significant positive association between mean ideological diversity and the correlation between the full and brief affective polarization measures ($\beta =\mathrm{>0.024}$, 95% CI [0.023,0.025], $p\<\mathrm{>0.001}$). At the mean level of party size, combinations covering a broader ideological spectrum achieve a higher construct validity (i.e., a $\mathrm{>2}SD$ increase in ideological diversity would amount to an increase in the Pearson correlation between the full and brief measure by about 0.05). Additionally, we find a statistically significant positive association with mean party size ($\beta =\mathrm{>0.067}$, 95% CI [0.066,0.068], $p\<\mathrm{>0.001}$), indicating that at the mean level of ideological diversity, the party size within a combination is positively related to its construct validity (i.e., a $\mathrm{>2}SD$ increase would roughly correspond to a 0.14 increase in construct validity).

Further, a statistically significant interaction between mean ideological diversity and mean party size, albeit with a very small effect size, suggests that the positive associations of ideological diversity and party size with construct validity are enhanced in combinations with larger average party size and more ideological diversity, respectively ($\beta =\mathrm{>0.006}$, 95% CI [0.005,0.007], $p\<\mathrm{>0.001}$). Figure 2 plots the predicted correlation between full and brief affective polarization measures as a function of mean ideological diversity and mean party size for combinations of 4 (Panel A1) and 5 (Panel A2) parties. To illustrate the metric, consider the following examples of four‐party combinations: While the party combination with the highest construct validity (VVD, PVV, GL, SP; $r_{construct}=\mathrm{>0.92}$) has a considerable ideological diversity (Standardized mean pairwise differences $=\mathrm{>1.55}$) and average party size (Mean vote share $=\mathrm{>0.13}$), the party combination with the lowest construct validity (PVV, FvD, SGP, DENK; $r_{construct}=\mathrm{>0.24}$) contains small (Mean vote share $=\mathrm{>0.05}$), ideologically homogeneous (Standardized mean pairwise differences $=-\mathrm{>0.59}$) parties. Online Appendix A2 lists additional party combinations with high/low construct validity.

Figure 2. Construct validity by ideological diversity and party size (weighted spread, NL).

Note: Panels A1 and A2 plot the predicted correlation between the full and brief affective polarization scores as a function of the mean ideological diversity and mean party size (+2 SD, +1 SD, mean, −1 SD, −2 SD) within samples of four and five parties (both z‐standardized), respectively, used to calculate the brief scores of affective polarization. Both predictors were z‐standardized. The red‐dashed line represents a correlation of $r=\mathrm{>0.80}$.

To assess the relationship between construct validity and mean ideological diversity and party size in the CSES, we employed a linear mixed model with the correlation between the full and brief affective polarization measures as the criterion, and the z‐standardized mean party size (i.e., vote share) and mean ideological diversity (i.e., pairwise ideological differences) of parties within a combination as well as their interaction as predictors. We also controlled for the number of parties included in the brief measures and, as the estimated variance for an additional random effect of the party system was close to zero, we used a simplified model that includes only a random intercept for countries. We find statistically significant positive associations between construct validity and mean ideological diversity ($\beta =\mathrm{>0.046}$, 95% CI [0.043,0.049], $p\<\mathrm{>0.001}$) and mean party size ($\beta =\mathrm{>0.152}$, 95% CI [0.149,0.154], $p\<\mathrm{>0.001}$). Given the multiplicative structure of our model, this indicates that at the mean level of party size and ideological diversity, respectively, both variables are positively related to construct validity. Further, the interaction effect suggests that those relationships are not conditional on the other variable, as the interaction term is very close to zero and not statistically significant ($\beta =\mathrm{>0.001}$, 95% CI [−0.002,0.003], $p=\mathrm{>0.560}$).

The positive association between construct validity and both ideological diversity and party size is also reflected in the ‘optimal' party combinations for each country that mostly encompass large parties that cover the whole ideological spectrum. For instance, for Germany the ideal combination contains ratings for the Social Democratic Party (SPD), the Christian Democratic Union (CDU), the Green Party (Gruene), and the Alternative for Germany (AFD) ($r=\mathrm{>0.93}$), while for Denmark it encompassed the Social Democrats (SD), Venstre (Liberal Party, V), the Danish People's Party (DF) and the Red‐Green Alliance (EL) ($r=\mathrm{>0.92}$). It is noteworthy that, despite the use of different data sources, the ‘optimal’ four‐party combination determined with the LISS data in the Netherlands (VVD, PVV, SP, GL) very closely matches the combination obtained in the CSES (VVD, PVV, SP, D66, $r=\mathrm{>0.93}$). Three out of four parties are identical and both combinations contain another progressive party as the fourth party. Further attesting to the robustness of our results, the combination identified with the LISS also exhibited high construct validity when evaluated with CSES data ($r=\mathrm{>0.86}$). In Online Appendix A6, we provide the optimal combinations for each of the studied countries.

Across 39 countries, we demonstrate that affective polarization measures derived from feelings towards three to five parties can have acceptable construct validity when they consist of large parties with sufficient ideological diversity. Our main findings and even specific party recommendations remain robust for alternative conceptualizations of affective polarization (i.e., unweighted spread, unweighted/weighted mean distance; Wagner, Reference Wagner2021). Detailed results are described in Online Appendices 1, 2, 4, 6, 8 and 11. We conclude that even with a substantial reduction in the number of parties, brief measures still capture what we intend to capture with the full affective polarization measures. In the next section, we will examine how shortening scales influences associations with frequently studied political correlates.

The predictive validity of brief measures

We assess predictive validity by examining bivariate correlations between the full/brief affective polarization measures and a set of often studied correlates. The correlates are scored such that positive correlations indicate that an increase in affective polarization is associated with an increase in the target variables. While we examine frequently studied outcomes of affective polarization, our design – like many other studies in this literature (e.g., Berntzen et al., Reference Berntzen, Kelsall and Harteveld2024; Gidron et al., Reference Gidron, Sheffer and Mor2022; Wagner, Reference Wagner2021) – does not allow for direct causal inferences.

We plot the mean correlations between the political correlates and full/brief affective polarization measures in the Netherlands (LISS) in panel A1 of Figure 3 (see Online Appendix 3.3 for more details). Consistent with existing literature (Harteveld, Berntzen, et al., Reference Harteveld, Berntzen, Kokkonen, Kelsall, Linde and Dahlberg2022; Harteveld & Wagner, Reference Harteveld and Wagner2023; Wagner, Reference Wagner2021), all correlations with affective polarization were positive.Footnote ⁴ Similar to the results for construct validity, the mean correlations remain relatively stable when different numbers of parties are included in the brief affective polarization measures. Notably, the truncation of the scores did not result in Type S errors when examining mean correlations as the signs of the correlations are the same when using brief or full measures. However, there are indications of Type M errors. For some constructs (e.g., turnout, identity strength), we see considerable declines in the correlations when comparing the full measure of affective polarization to brief measures based on four or five parties. For turnout, for instance, the full affective polarization score correlates at $r=\mathrm{>0.28}$, while mean correlations with measures derived from four or five parties are $M_r=\mathrm{>0.19}$ and $M_r=\mathrm{>0.21}$, respectively.

Figure 3. Predictive validity – brief affective polarization measures (weighted spread).

Note: Bivariate correlations between the full and brief affective polarization measures (weighted spread of like/dislike) and frequently studied political correlates. Panels A1 and B1 show the mean correlations for all possible combinations of a given number of parties in the Netherlands (LISS, A1) and across 39 countries (CSES, B1). Panel A2 shows the correlations for the optimal party combinations for a given scale length in the Netherlands (LISS), selected based on construct validity. Panel B2 shows the mean correlations for the optimal party combinations for a given scale length (based on construct validity) across the 39 countries in the CSES dataset. Panels B1 and B2 are divided based on the highest number of parties included in the full affective polarization measures as per the availability for each country in the CSES data. Further details (e.g. $SE$, 95% CIs) can be found in Online Appendices 3.3. and 9.2.

In Figure 1, we demonstrated substantial variability in the construct validity of brief affective polarization measures of the same length. This implies that associations with other variables may also not be uniform across brief measures of the same length that are based on different party combinations. Looking at four‐party combinations, we observe a wide range of correlations with satisfaction with democracy ($rang{e}_r[-\mathrm{>0.23};\mathrm{>0.27}]$), confidence in democracy ($rang{e}_r[-\mathrm{>0.22};\mathrm{>0.28}]$), party identification ($rang{e}_r[\mathrm{>0.01};\mathrm{>0.20}]$), identity strength ($rang{e}_r[\mathrm{>0.02};\mathrm{>0.29}]$), political interest ($rang{e}_r[\mathrm{>0.01},\mathrm{>0.21}]$) and turnout ($rang{e}_r[\mathrm{>0.04};\mathrm{>0.28}]$). It is apparent that, despite stable mean correlations across the brief affective polarization measures, the application of individual brief measures with specific party combinations can lead to notable variations in magnitude (Type M error) and even sign (Type S error) of associations. This is unsurprising given that brief measures calculated from representative, ideologically diverse sets of parties seem to mirror full measures in capturing affective polarization (i.e., high construct validity), while measures focusing on small, uniform parties may reflect different constructs (i.e., low construct validity) and, hence, associations.

To ascertain whether party combinations with high construct validity also yield an enhanced predictive validity, we examine the correlations between the ‘optimal’ party combinations of a given length, as determined by their construct validity, and the political correlates in panel A2 of Figure 3. The results demonstrate remarkable consistency between the full and ‘optimal’ brief measures in their associations with political correlates across the entire length of the scale, with only minimal variation. This emphasizes that by carefully selecting party combinations for construct validity, we can also achieve a good predictive validity with brief affective polarization measures.

We repeated this approach across the 39 countries included in the CSES data. Mean correlations across different party systems and numbers of parties included in the brief affective polarization measures are plotted in panel B1 of Figure 3. Again, all correlations with affective polarization were positive. In each party system, we find considerable stability of the mean correlations across the different numbers of parties used to calculate affective polarization, up until a certain number of included parties that aligns well with the previously identified points at which the mean construct validity begins to decline (i.e., three parties in five‐ or six‐party systems, and four parties in seven‐, eight‐ or nine‐party systems). However, again, we find substantial variations in the correlations between brief affective polarization measures and target variables across different combinations of the same number of parties within a country (see Online Appendix 9.2). Thus, we again explored if selecting combinations based on high construct validity also improves the predictive validity. To assess this, we selected the party combinations with the highest construct validity for each length of the brief affective polarization measures within each country. Then, we calculated the mean correlations of these optimal combinations with the target variables for each length of the affective polarization measure within each party system. The results are displayed in panel B2 of Figure 3. Again, the correlations are strikingly consistent across the full measures and ‘optimal' brief affective polarization measures and there is only minimal variation across the length of the brief measures.

In line with our findings on construct validity, we observe that brief affective polarization measures, based on feelings towards only three to five political parties, can exhibit acceptable predictive validity when the parties are selected based on high construct validity. This alignment between findings for both metrics also suggests that there is no trade‐off between achieving higher construct validity and maintaining strong predictive validity. We find this consistently across both the LISS and CSES datasets as well as across four different operationalizations of affective polarization (details on other measures are provided in Online Appendices 1.3, 2.3, 4.3, 6.2, 8.2 and 11.2).