ARE HIGHLY ENGAGED EXPERTS ALSO MORE PESSIMISTIC?

Unlike existing expert surveys on democracy across countries (e.g., V-Dem, Freedom House, and Polity), our expert pool is drawn entirely from American universities. Our mailing list includes approximately 10,000 unique email addresses, and typically 500 to 1,000 respondents complete the surveys. In any given wave, about 5% to 10% of the discipline participates in our surveys; thus, BLW’s expert surveys are completed by a much larger pool of experts than any of the existing global measures of democracy.Footnote ²

Given how BLW recruits expert respondents, self-selection is a possible source of bias. If experts who participate in our surveys are more concerned about the state of US democracy than those who do not participate, our measures could be systematically overstating the extent to which the discipline as a whole perceives threats to American democracy. Moreover, if political scientists indeed are selecting in and out of participation on the basis of their level of concern, then respondents who participate sporadically—or even only once—should be less pessimistic than those who participate regularly, thereby constituting a disproportionate share of responses in any given survey.

To test this, we exploited the fact that each BLW expert respondent was assigned a unique participant ID that is stable across survey waves.Footnote ³ For each survey wave, we computed the mean evaluation of US democracy on a 100-point scale among respondents who participated in only that one survey wave, respondents who participated in a total of two waves, respondents who participated in three waves, and so forth. In figure 1, each facet shows the mean rating on the 0–100 scale item among respondents who participated in 1, 2, 3…14 surveys of the 18 survey waves overall.Footnote ⁴ Contrary to the self-selection hypothesis, figure 1 shows no pattern by which frequent responders are more or less sanguine about US democracy than infrequent responders.Footnote ⁵

Figure 1

Ratings of American Democracy in Waves 2–5, by Expert Participation

Horizontal lines show unconditional means by wave. Point size represents the number of respondents.

MEDIA IMMERSION

Little and Meng suggest that immersion in a media environment that is pessimistic about democracy might generate expert-coder bias. Media sources increasingly indicate that democracy is under threat. Democracy experts bathe in this discourse and react to it. The same experts also generate the subjective assessments used to measure erosion. If all of this is the case, then we might expect to see that experts who are more heavily marinated in democracy-alarmist media are particularly pessimistic about democracy. We call this the consumption hypothesis. We agree with Little and Meng that the media-consumption narrative is plausible, although we struggle to imagine a research design that provides a rigorous causal test. Moreover, we note that much of the media coverage about threats to democracy is rooted in the work of academics themselves. As such, media coverage should not be taken as an exogenous force to which and by which political scientists and other experts happen to be exposed and influenced. Rather, at least part of the concern expressed by the academic world is causally prior to media coverage.

A second possibility is that experts have balanced information diets that neither overstate nor understate the threat to democracy but that the bundle of information that experts produce for public consumption (e.g., academic articles, op-eds, and social media posts) is disproportionately of the “threat-to-democracy” genre. We call this the production hypothesis.Footnote ⁶

Data from the 17th wave of BLW surveys, fielded in October 2022, touch on this theme of skewed scholarly production. In an attempt to detect any selection effects that may skew the public face of scholarship on the state of democracy, we asked our expert sample whether and how often they use Twitter. We also asked how often respondents tweeted about issues related to the state of American democracy. Twitter is a relevant platform for this test because of its role as a digital “town square” for academic communities, including the political science community (Bisbee, Larson, and Munger Reference Bisbee, Larson and Munger2022). It is plausible that excessive pessimism regarding the state of American democracy may attract a substantial audience, given the propensity to consume negative news (Robertson et al. Reference Robertson, Pröllochs, Schwarzenegger, Pärnamets, Van Bavel and Feuerriegel2023; Sacerdote, Sehgal, and Cook Reference Sacerdote, Sehgal and Cook2020).

In our October 2022 survey of 682 political scientists, 40% of the 626 who answered the question reported that they did not use Twitter at all (i.e., non-users), 49% used the application but did not tweet regularly about American democracy (i.e., Twitter consumers), and 11% tweeted about American democracy once a week or more (i.e., democracy tweeters). We asked all of the experts to rate the then-current performance of US democracy on a 100-point scale and also to make projections five and 10 years into the future. If the production hypothesis is correct, we expected to see a group of highly concerned experts who select into frequent public discussions of the state of US democracy and who skew public-facing scholarship in the direction of alarmism. That is, we expected our democracy-tweeter experts to assess American democracy more negatively than those who are “less online.” Democratic pessimism among the Twitter consumer experts would be consistent with the consumption hypothesis. Figure 2 illustrates the democracy ratings as of October 2022 as well as future assessments of each group.

Figure 2

Expert Ratings of Democracy by Twitter Usage

Vertical error bars are 95% confidence intervals.

On current assessments of democratic performance, the mean rating among non-users was 65; among Twitter consumers it was 68; and among democracy tweeters it was 66. The difference between Twitter consumers and non-users reached conventional statistical significance (p=0.04), with consumers more optimistic than non-users—contrary to the consumption hypothesis. Projecting into the future, all three groups anticipated democratic erosion. At five and 10 years out, Twitter consumers were the most optimistic, followed by non-users, and democracy tweeters were more pessimistic. Differences between non-users and either of the Twitter-engaged groups never reach statistical significance. The differences between Twitter consumers and democracy tweeters were significant at p=0.04 and p=0.03, respectively. However, we cannot imagine any theoretical account by which a limited amount of Twitter immersion should cause democratic optimism whereas further increasing Twitter immersion should cause pessimism. In summary, the differences that we observed in both current assessments and future projections did not map onto an account by which more Twitter immersion produces more pessimism.

It is important to note that the data presented so far are drawn from survey questions not purposely designed to test Little and Meng’s proposition about expert bias. Rather, Little and Meng’s intervention opened an important discussion and prompted us to reexamine data collected for other purposes, seeking leverage on this new debate. There could be a pessimism effect big enough to cause the observed decline in certain countries’ V-Dem polyarchy indices but one limited enough to evade our imperfect searchlight. We note that the more dire claims about the state of democracy in recent V-Dem reports (e.g., Boese-Schlosser et al. Reference Boese-Schlosser, Alizada, Lundstedt, Morrison, Natsika, Sato, Tai and Lindberg2022) rely on shifting the unit of analysis from the country to the individual citizen. Thus, recent declines in V-Dem’s polyarchy scores for a few large countries (in particular, India, where 17.7% of the world’s population lives) overshadow democratic improvements in smaller countries. As Little and Meng observe, the average V-Dem polyarchy score across countries has remained fairly constant since 1990—in fact, the trend line is remarkably similar to that of Little and Meng’s proposed alternative measures. Nonetheless, we still may be concerned that a pessimism bias is driving or exaggerating the appearance of backsliding in the countries where V-Dem has registered declining polyarchy scores.

In our most recent BLW survey, conducted in June–July 2023, we sought to determine more directly whether the specific political scientists who generate expert assessments on which contemporary democracy scholarship is based are systematically biased toward democratic pessimism.Footnote ⁷ Specifically, at the end of a BLW expert survey, we included questions—designed in collaboration with Little and Meng—that asked respondents whether they had ever been invited to serve as a coder for V-Dem. We also asked whether they had served (if invited) and about their willingness to serve (if not invited). Of the 544 expert respondents who completed this section of our survey, 484 (89%) had never been invited to serve as V-Dem codersFootnote ⁸; 16 (3%) had been invited but did not participate; and 44 (8%) had been invited and served as coders.

Prior to being asked about V-Dem participation, our expert respondents had rated on a 100-point scale the quality of democracy in a random subset of six countries around the world in addition to the United States from the following: Brazil, Hungary, India, Italy, Israel, Kenya, Mexico, Peru, the Philippines, Poland, Turkey, and the United Kingdom.

Expert-coder bias could operate through either V-Dem invitations (i.e., experts who are invited are more pessimistic than those who are not invited) or self-selection (i.e., experts who choose to participate are more pessimistic than those who decline). Figure 3 shows the average ratings for each country. The left panel compares ratings from expert respondents in our sample who had and had not been invited by V-Dem. The right panel compares ratings from those willing versus those unwilling to participate with V-Dem.

Figure 3

Invitation Bias and Participation Bias Among V-Dem Coders

We find no evidence of bias toward pessimism at either stage of selection into the V-Dem coder pool, invitation or participation. The average democracy ratings among those who were invited to code for V-Dem were higher than the average ratings among uninvited experts for 12 of 13 countries (except only Turkey). The difference reached statistical significance for Brazil and Peru. With regard to participation, the average ratings were higher among those willing to code for V-Dem than among those who were unwilling for all 13 countries, with statistically significant differences for the United States, Brazil, Mexico, and Poland. Overall, those political scientists whom V-Dem targets and those inclined to participate if asked appeared more, not less, sanguine about democracy around the world than experts outside of that V-Dem coder pool.

There are three important caveats to our analytical strategy. First, those experts who code do so for specific countries for which they have the greatest expertise. Unfortunately, our survey sample does not provide a sufficient number of “direct hits” (i.e., V-Dem coders who also rated their specific country of expertise in our survey) to allow comparison with ratings from the broader set of political scientists. Second, our sample of political scientists may be vulnerable to self-selection bias; it is possible that experts who declined to take part in our survey hold systematically different attitudes from those who did.Footnote ⁹ Third, it also is possible that political science as a whole is unduly pessimistic about democracy across the world, in which case the baseline against which we are comparing V-Dem coders does not reflect a ground truth about democracy around the world. If such a bias were new or increased in recent years, it could cause a universal shift in coder standards—according to Little and Meng’s formulation—that our approach might not detect.

EXPERT VERSUS PUBLIC ASSESSMENTS OF DEMOCRACY

In addition to our expert surveys, BLW also routinely polls the general public about the state of democracy in the United States and, occasionally, other countries. Comparing recent rankings of democracy in 13 countries, expert assessments about democratic performance exhibit far less compression than those of the public and offer more precise estimates (with lower variance as a group)—as we would expect and hope if the experts are actually better informed.

Figure 4 shows mean democracy ratings on the 100-point scale, with 95% confidence intervals, for 13 countries plus the United States as of October 2022. The rank ordering of countries by experts and the public is almost identical, with North Korea at the bottom and Canada at the top. However, mean expert assessments range from 2 to 84, whereas the public’s assessments range from 18 to 59. The expert assessments are far more precise, with country-level standard deviations ranging from 6.7 (North Korea) to 20.7 (Israel), compared with 21.8 (Great Britain) to 26.6 (Israel) for the public.

Figure 4

Expert and Public Ratings of Democracy in 13 Countries

Horizontal error bars are 95% confidence intervals. Data are from Wave 17 (October 2022) of BLW surveys.

Returning our focus to the United States, figure 5 shows the time series from 2017 through 2022 of mean responses with 95% confidence intervals on the same 100-point scale from our expert respondents (shown in green) and the public (shown in purple).Footnote ¹⁰

Figure 5

Expert and Public Ratings of US Democracy, 2017–2023

Vertical error bars are 95% confidence intervals.

If Little and Meng’s thesis about expert bias is correct, we might expect experts to be more pessimistic than the public or at least that the relative optimism of experts, compared to the public, would have declined. In fact, relative to the public, our experts consistently rate American democracy approximately 10 points higher. Neither do we observe any evidence that experts are increasingly pessimistic, relative to the public as a benchmark, over time. Indeed, BLW’s most recent survey, conducted in June–July 2023, shows the largest optimism gap yet between experts and the public.

Next, we consider how expert ratings compare to those of the public across a range of democratic principles. BLW regularly surveys both groups on 30 principles related to elections and voting; citizen rights and protections; and accountability, institutions, and norms. A full description of each principle is included in the Appendix. Figure 6 illustrates the proportion of experts (green circles) and the public (purple squares) who, in June–July 2023, rated the United States as fully or mostly meeting each standard.

Figure 6

Expert and Public Ratings of 30 Indicators of US Democratic Performance

Horizontal error bars are 95% confidence intervals. Statements are in descending order of performance ratings for experts surveyed in June–July 2023.

The pattern of expert discernment and precision relative to the general public is similar to what is observed in figure 4. Expert assessments range from 9% (i.e., districts not biased) to 90% (i.e., government statistics not politically influenced). The range in these responses highlights the value of expert-coded measures of specific, concrete variables. Expert ratings of American democracy tend overall to cluster between 65 and 70 on the 100-point scale. However, when the experts are asked about specific components, their assessments range from widespread confidence (e.g., in election integrity and many civil liberties) to almost consensus on principles not met (e.g., unbiased districts or adherence to norms of mutual respect and cooperation). By contrast, the public’s assessments across the 30 standards of performance are relatively clustered. Reviewing the percentage of respondents who agree that the United States fully or partially meets each standard, the range is from 15% to 56%, with 20 of the 30 items falling between 25% and 50%.Footnote ¹¹

One interpretation of figure 6 is that many respondents in our public sample may have a general sense of the state of democracy and work backward from that overall rating to assess the individual indicators of performance on which we query them. There are notable exceptions, particularly regarding issues for which there have been salient elite cues, such as fraud-free elections and equal voting rights. Nonetheless, the broad pattern appears to be reasoning from the general to the specific. By contrast, the experts—informed and opinionated—address each standard independently.

EXPERT ASSESSMENTS AND PARTISANSHIP

A final possible source of bias in expert assessments (albeit one that was not directly raised by Little and Meng) is that the BLW expert sample may be systematically more aligned with one partisan group compared to another. BLW does not ask our expert respondents about their partisanship, but we do ask respondents in our public sample.Footnote ¹² Figure 7 shows the correlation coefficients between three pairs of subgroups (i.e., experts and Democrats, experts and Republicans, and experts and partisan independents) across the 30 performance indicators for each survey as far back as October 2017.Footnote ¹³ A few patterns are worth noting.

Figure 7

Correlation Between Experts and Partisan Groups on 30 Indicators of Democratic Performance, by Wave

Figure shows the correlation coefficient in the proportion of experts and different partisan groups that rate the performance of US democracy on 30 different indicators as positive. Each point is the correlation coefficient for a given wave, across those 30 indicators.

First, assessments among our expert sample consistently align with those of Democrats more than with Independents and far more than with Republicans. The correlation coefficients between expert and Democratic assessments on the 30 principles hover around 0.80, with no dramatic change over time.

The second broad pattern is that what began as a moderate alignment between expert and Republican assessments decreased drastically beginning around Fall 2019, which coincides with the process that led to the first impeachment of then-President Donald J. Trump over his pressuring of Ukraine to investigate the Biden family. A second even more precipitous decrease followed in late 2020, coinciding with the immediate aftermath of the 2020 election (and Trump’s second impeachment). The sharp divergence between expert and Republican assessments largely derives from growing pessimism among Republicans and optimism among experts after the 2020 election. Republicans became more concerned about election fraud and free-speech protections; experts became less concerned about politically motivated investigations and the use of government agencies to monitor and attack political opponents. On one measure—whether politicians publicly concede defeat—experts became much more concerned than Republicans after the 2020 election.

The Republican–expert correlation rebounded sharply in November 2022, in the wake of the midterm elections, and remained approximately level in June–July 2023. This shift corresponds to increasing confidence among Republicans in fraud-free elections (recovering from a low of 18% to 35%) and increasing confidence among experts that politicians concede defeat (recovering from a low of 34% to 57%).

Third, the correlation between partisan Independents and our experts, which held steady into 2020, eroded more gradually during the next two years—although never as sharply as among Republicans—and partially recovered in late 2022 and 2023.

There are at least two competing interpretations of these patterns. The more obvious is that our sample of experts likely skews Democratic. This is unsurprising because American university faculty members are well known to skew overwhelmingly Democratic (Langbert and Stevens Reference Langbert and Stevens2021). This predates the beginning of our time series and continues throughout the study. Another possibility—albeit one that is difficult to establish systematically—is that the partisan groups differ in the soundness of their assessments of performance on our 30 democratic principles, with Democrats being more accurate than Republicans and Independents. We note that the Democrat–expert correlation remains steady throughout, whereas high-salience political events since 2019 appear to have breached any common ground that our experts shared with Republican partisans on democratic performance.