Selection of relevant measures of democracy

Since all existing measures of democracy cannot be effectively compared and analysed in a single study, I begin my enquiry by selecting some of the most currently relevant measures of democracy. The selection is based on academic importance, measurement scale, availability, and coverage. Dichotomous indicators such as Alvarez et al. (Reference Alvarez, Cheibub, Limongi and Przeworski1996) Democracy and Dictatorship are excluded, so that we capture not only the distinction between autocracy and democracy, but also different degrees of democracy. Ordinal measures that are hardly ever treated as interval such as Skaaning et al. (Reference Skaaning, Gerring and Bartusevičius2015) Lexical Index of Electoral Democracy are excluded as well. Finally, measures that have limited time coverage such as Economist Intelligence Unit’s (2019) Democracy Index have not been updated regularly in recent years such as Coppedge et al. (Reference Coppedge, Alvarez and Maldonado2008) Contestation and Inclusiveness, or rely explicitly on other measures such as Kaufmann et al. (Reference Kaufmann, Kraay and Mastruzzi2011) Voice and Accountability are left out from the analysis.

To assess the academic importance of measures of democracy, I use Coppedge et al. (2017) study, which reports the number of Google Scholar citations of several democracy indices. Center of Systemic Peace’s Polity IV leads the pack with nearly 5000 citations. Freedom House’s Civil Liberties and Political Rights are the second most “impactful” measures of democracy with more than 1,500 citations on Google Scholar. Vanhanen’s Competition and Participation are the next most cited measures of democracy that fulfil our selection criteria. The three above-mentioned measures have been widely used in political research, cover nearly all countries around the world over many decades, and are publicly available. Besides these three well-established measures, I include also the more recently developed V-Dem’s Polyarchy index, which has gained widespread recognition the last few years. Thus, to sum up, this article analyses four measures of democracy: Polity IV (Marshall et al. 2019), Vanhanen Index (Vanhanen 2019), Freedom Rating (Freedom House 2019a), and V-Dem Polyarchy (Teorell et al. Reference Teorell, Coppedge, Lindberg and Skaaning2019). Table 1 presents an overview of these indices.

Table 1:

Main characteristics of frequently used measures of democracy

Note: Number of countries refers to year 2017

Polity IV (PIV) was originally built upon Gurr’s (Reference Gurr1974) and Eckstein and Gurr’s (Reference Eckstein and Gurr1975) studies on political systems, with the objective of analysing whether common beliefs about the characteristics of state authorities have possible historical or cross-cultural patterns around the world. The Polity IV index is composed of the Institutional Democracy and Institutional Autocracy sub-indices, which measure, respectively, the democratic and the autocratic features of a country. The two sub-indices, which in turn are based on several more specific sub-components, are ultimately combined into the Polity IV index. The index has a 21-point scale from more autocratic to more democratic and provides annual data for all countries in the world with a population of at least 500,000 since 1800.

Vanhanen’s index of democratisation (VAN) measures democracy as a two-dimensional construct formed by the amount of electoral competition between parties and the amount of voting turnout in the elections (Vanhanen Reference Vanhanen2000). The index aggregates two equally weighted sub-indicators, Electoral Competition and Political Participation, which are both based on “hard” quantitative electoral data. Thus, according to Vanhanen (Reference Vanhanen2000) the index provides more objective information than many other democracy indices. In theory, Vanhanen’s index ranges from 0 to 100 (from low to high democracy), although, in practice, no country has ever received a score higher than 49. Annual data is provided for most countries in the world since 1810.

Freedom Rating (FH), developed and produced initially by regional studies specialist Gastil, is conceptually rooted in the principles of the Universal Declaration of Human Rights of 1948 (Freedom House 2019b). The index aggregates two main sub-indices, Political Rights and Civil Liberties, which in turn are based on several more specific sub-indicators. The final Freedom Rating score of a country is the average of these two main sub-indices. Annual data is available since 1972 for virtually all existing countries and some territories (e.g. Hong Kong, West Bank) in the world. The index measures democracy on a scale from 1 to 7, where a lower score stands for a higher level of democracy.

V-Dem’s Polyarchy index (VDEM), rooted in Dahl’s (Reference Dahl1971) influential theory of democracy, quantifies democracy as a combination of “freedom of association, suffrage, clean elections, elected executive, and freedom of expression” (Coppedge et al. 2017: 10). In V-Dem project’s conceptual scheme, the Polyarchy index is conceived to measure the fundamental aspects of democracy (Coppedge et al. 2017). The index synthesises 40 sub-indicators that are coded by country experts (mainly academics) and research assistants (Teorell et al. Reference Teorell, Coppedge, Lindberg and Skaaning2019). Polyarchy ranges on a continuous scale from 0 to 1, from low to high. Annual data covers nearly all countries of the world since 1789.

Research strategy

I start the quantitative analysis by exploring the statistical features and evaluating the distributional characteristics of the selected indices through violin plots. While we should not expect measures of democracy to follow a Gaussian distribution by nature, we would prefer—for statistical purposes—to have measures that are more or less normally distributed, for the simple reason that many common statistical analyses and tests assume such a distribution. An analysis of the distributions is an informative task per se. Yet, it evidences also some more subtle weaknesses in the measures that are further discussed. In particular, the ability to differentiate between countries (Knutsen Reference Knutsen2010), especially at the extremes of a scale (Goertz Reference Goertz2020), is a desirable property of democracy measures. Violin plots are especially helpful in detecting bumps within a distribution (Hintze and Nelson Reference Hintze and Nelson1998), and as we shall see, some of the democracy measures tend to clump at extreme values. Such scales should be further extended to capture all interesting variation (Goertz Reference Goertz2020).

Second, after analysing the statistical features and distributions of the data, I run bivariate correlations to assess the strength of the linear associations among our measures of democracy. Since all measures quantify democracy, we expect them to be strongly related to one another. To get a general view of the associations, correlations are computed between pairs of democracy measures using all available observations since 1972.Footnote ¹ To examine whether these relationships differ from year to year and/or over time, correlations are run also at 5-year intervals. These correlations provide information about the statistical association and the convergence of the measures. If currently relevant measures of democracy are strongly associated among one another, we can conclude that their convergence is high.

Finally, through a battery of regression models, I analyse whether the choice of the measure of democracy affects the interpretation of the results, or in other words, whether measures of democracy are interchangeable or not. Following previous empirical literature on measurement interchangeability (e.g. Casper and Tufis Reference Casper and Tufis2003; Møller and Skaaning Reference Møller and Skaaning2011; Högström Reference Högström2013), I regress our measures one by one on the same set of predictors, keeping everything else equal. According to the interchangeability criterion, “equivalent measures should produce similar causal inferences” (Seawright and Collier Reference Seawright and Collier2014: 124). Hence, if the conclusions are not determined by the choice of the measure, we can claim that measures of democracy are interchangeable. To exclude selection bias, I run the regressions with only country-years common to our four measures of democracy.

The democracy indices are regressed on a set of common predictors of democracy based on both Högström’s (Reference Högström2013) and Casper and Tufis’ (Reference Casper and Tufis2003) studies. These common or “structural” predictors are economic development, growth, trade openness, inflation, population, education, presidentialism, parliamentarianism, and party fractionalisation. The first five variables are taken from the World Development Indicators dataset (World Bank 2017). Economic development is measured by the natural logarithm of GDP/capita in constant 2010 USD. Growth is measured by the annual GDP/capita growth (as a percentage). Trade openness is measured by the sum of exports and imports of goods and services as a share of GDP. Inflation is measured by annual change in consumer prices (as a percentage). Population is measured by the natural logarithm of the estimated total population. Education is measured by the average years of education among citizens older than 15 and is retrieved from the Varieties of Democracy dataset (Coppedge et al. 2019). Party fractionalisation is measured by the probability that two random draws from the lower legislative chamber will be from the same party and is taken from Henisz’s (2017) POLCON dataset. Presidentialism and parliamentarism are dummy variables coded according to Anckar and Fredriksson’s (Reference Anckar and Fredriksson2019) classification of political systems.

The selected set of variables allows us to assess the interchangeability of measures of democracy with a sample of nearly 130 countries from 1973 to 2017. To cope adequately with problems common to panel regressions in cross-national contexts, as recommended by Beck and Katz (Reference Beck and Katz1995), I estimate the models with ordinary least squares (OLS) and panel-corrected standard errors (PCSE), which correct for heteroskedasticity and contemporaneous correlation. Serial correlation is corrected with a lagged dependent variable on the right-hand side of the equation and problems of reverse causality are mitigated by lagging all independent variables by 1 year. Throughout the empirical analysis, to facilitate comparison, measures of democracy are normalised (min–max) and transformed to range from 0 (low) to 100 (high). FH is reversed so that higher scores indicate more democracy.

Main statistical properties

Violin plots in Fig. 1 present the main statistical properties of our four measures of democracy in all common country-years from 1972 to 2018. The black rectangle in the middle of each “violin” represents the median, the white box around it represents the interquartile range, and the whiskers show the distribution of the remaining values, with the exception of possible outliers. The grey area around the white box illustrates the shape of the distribution at various levels of democracy.

Fig. 1

Violin plots of measures of democracy (1972–2018)

Overall, none of the selected measures seems to provide anything close to a fairly normal distribution of values. VAN has a substantially lower median value compared to the other measures and its distribution is heavily skewed to the right. In practice, as shown by the high frequency of observations at the bottom of the scale, this means that a large amount of country-years in VAN are coded as highly autocratic. On the contrary, the other three indices of democracy have a more or less pronounced bimodal distribution, with peaks near the two ends of the scale and less observations in the middle of the scale. Intermediate levels are particularly empty with PIV, as shown by its comparatively large interquartile range. The very extremes are more clumped with FH and PIV than with VDEM. FH’s median is precisely in the middle of the scale, PIV’s median is closer to the democratic end of the scale, and VDEM’s median is closer to the autocratic end of the scale.

Besides showing that measures of democracy are far from being Gaussian, the distributional characteristics reveal some interesting differences among the measures. VAN seems to be unable to capture variation at the bottom extreme of the scale, as confirmed by the fact that as many as 25.7% of all country-years from 1972 to 2018 have the minimum score with VAN. Only in 2000, there are nearly 30 countries with the minimum possible level of democracy with VAN. Since the other measures are able to tell the difference between almost all these countries (VDEM) or at least between many of them (PIV and FH), such an agglomeration of country-years at the bottom extreme of the scale with VAN is more likely to reflect an excessive compression rather than real equivalence between the observations. Furthermore, since all these countries have the lowest possible score in VAN, the index is not able to capture any decrease in the level of democracy in suspiciously many countries.

To a lesser extent, PIV and FH suffer also from similar distributional shortcomings at the very extremes of the scale. Both indices compress several countries especially at the maximum level of democracy, suggesting that the scales of PIV and FH should be further extended beyond their current upper extremes to capture a more nuanced view of highly democratic countries. In total, from 1972 to 2018, 18.6% of country-years in PIV and 14.1% country-years in FH are coded with the maximum score, and for instance, both indices are unable to distinguish between more than 30 countries with maximum scores each year from 2010 to 2015. By construction, any possible increase in democracy in these countries has been left unnoticed by both PIV and FH. Furthermore, as before, we can confidently assume that such an agglomeration of observations reflects excessive compression instead of true equivalence, because both VAN and VDEM are able to distinguish between the vast majority of these countries.

The above discussed violin plots show that none of the measures of democracy is normally distributed. Heavy non-normality is not problematic in itself but becomes limiting if users of these measures wish to employ them in statistical analyses and tests that assume normally distributed data. The evaluation of the distributional characteristics of the data shows as well that some of the measures fail to capture variation between non-democratic cases at the bottom end of the scale (VAN) or between highly democratic cases at the top end of the scale (PIV and FH). Such an artificial agglomeration of cases at the extremes of the scale can be considered as a shortcoming of VAN, FH, and PIV. From this perspective, VDEM is better constructed than the other measures.

Convergence

Table 2 reports bivariate correlations among our four democracy indices, when all available observations across countries and over years are taken into account. The bottom-left quadrant reports Pearson’s correlation coefficients, while the upper-right quadrant reports Spearman’s rank correlation coefficients. Both methods are used to add robustness to the results. Pearson’s correlations measure the strength of linear relationships and are more sensitive to outliers, while Spearman’s correlations measure the strength of monotonic relationships and are less sensitive to outliers. The choice of the estimator, however, does not alter substantially the results.

Table 2:

Pairwise correlation coefficients of measures of democracy (1972–2018)

Note: Upper-right quadrant: Spearman’s correlations; bottom-left quadrant: Pearson’s correlations; number of observations in parentheses; all coefficients significant at the p < 0.001 level

All bivariate correlation coefficients, regardless of the chosen method, are higher than 0.80, indicating high convergence among measures of democracy. With Pearson’s method the weakest correlation is between VAN and PIV (0.82), while the strongest is between VDEM and FH (0.92). With Spearman’s method, instead, the weakest correlation is between VAN and FH (0.84) and the strongest is between VDEM and PIV (0.92).

In general, the high convergence of measures of democracy is confirmed by Fig. 2, which illustrates bivariate Pearson’s correlation coefficients among our indices at 5-year intervals, from 1975 to 2015. Despite being strong, however, these correlations are not always constant over time. For instance, the bivariate correlation between FH and PIV changes minimally from one period to another (0.88 in 1975 and 0.86 in 2015), whereas the bivariate correlation between VAN and VDEM changes considerably over time (0.92 in 1975 but only 0.77 in 2015). In most cases, measures of democracy have become less similar among one another, as shown by the predominantly decreasing lines in the plot. Heterogeneity among correlation coefficients has increased over time. In 1975, the correlation coefficients among measures of democracy range only between 0.84 and 0.92, but in 2015, they range no less than from 0.76 to 0.95.

Fig. 2

Evolution of correlations over time: line plot of linear bivariate correlations among measures of democracy in 5-year intervals, 1975–2015

The findings of this section show that, overall, measures of democracy are convergent and strongly associated among one another. We would expect the four indices to be highly interchangeable, even if we have seen as well that their convergence has mostly decreased in the last decades. It seems that the wave of democratisations that begun in the mid-1970s has not been captured in the same way by the measures. To sum up, VAN has become considerably less associated with the other measures. Likewise, PIV has become less associated with the other measures, but not as much as VAN. The only exception is the relationship between FH and VDEM, which has become slightly stronger from 1975 to 2005 and remained relatively stable from 2005 on.

Interchangeability

Table 3 presents the results of the first set of longitudinal regressions of measures of democracy on structural predictors. All other things equal, the results show that different measures of democracy can lead to substantially different findings and interpretations. According to this first set of models, frequently used measures of democracy are not interchangeable at all.

Table 3:

Longitudinal regressions of democracy on structural predictors (1973–2017)

Note: T-values in parentheses; *p < 0.05, **p < 0.01, ***p < 0.001. Constant coefficient measured, but not reported. All models are estimated with OLS-PCSE and include a lagged dependent variable (not reported) on the right-hand side of the regression equation. Independent variables are lagged by 1 year. The years 1973–2017 refer to the dependent variable

The only similarities are that GDP/capita growth, inflation, and population are not significant predictors of democracy in any of the models. All the other associations depend meaningfully on the chosen measure of democracy. GDP/capita is a positive and significant predictor of democracy only when democracy is quantified with VAN (at the 99.9% level), VDEM (at the 99.9% level), or FH (at the 95% level). On the contrary, when democracy is quantified with PIV, there is no significant relationship between GDP/capita and the level of democracy. Trade openness is significantly and inversely related to democracy with PIV (at the 95% level) and VDEM (at the 99% level), but with VAN and FH the relationship is not significantly different from zero. Education has a significant positive effect on democracy with VAN, whereas with PIV, FH, and VDEM there is no significant relationship between education and democracy. Presidential systems are significantly and positively related to democracy only with VAN and VDEM (in both cases at the 95% level). Parliamentary systems are significantly and positively related to democracy only with FH (at the 95% level) and VAN (at the 99.9% level). There is a significant and positive relationship between party fractionalisation and democracy only with PIV and VAN. With FH and VDEM, instead, the relationship between party fractionalisation and democracy is not even close to conventional levels of statistical significance.

The findings of the first set of regressions show unquestionably that measures of democracy are weakly interchangeable in our full sample of common country-years. Nevertheless, since we previously found that in general the convergence among measures of democracy has become weaker over time, I investigate whether the interchangeability of the measures follows a similar pattern as well. I divide the dataset into two similar parts (1973–1995 [T = 23]; 1996–2017 [T = 22]) and regress each measure of democracy on our selection of structural predictors. Due to our earlier findings, we would expect the measures to be more interchangeable between 1973 and 1995 than between 1996 and 2017.

The expectations are confirmed by the regression estimates. In the first sample of years (1973–1995), the estimated results lead to similar interpretations nearly without any exceptions (Table 4). Only the effect of education depends meaningfully on the chosen measure of democracy: education is significantly and positively related to democracy with VAN and FH, but has no significant relationship with democracy with PIV and VDEM. No other prediction is statistically different from zero at conventional levels. This means that, in general, in the first sample of years (1973–1995), the choice of the measure does not play a decisive role in generating different results. In fact, from 1973 to 1995, PIV and VDEM lead exactly to the same conclusions across all predictors. Given these results, it would not be surprising to find that the weak interchangeability of the full sample (1973–2017) is driven by increased divergencies among measures in more recent years (1996–2017).

Table 4:

Longitudinal regressions of democracy on structural predictors (1973–1995)

Note: T-values in parentheses; *p < 0.05, **p < 0.01, ***p < 0.001. Constant coefficient measured, but not reported. All models are estimated with OLS-PCSE and include a lagged dependent variable (not reported) on the right-hand side of the regression equation. Independent variables are lagged by 1 year. The years 1973–1995 refer to the dependent variable

As suspected, in the second sample of years (1996–2017) the choice of the measure of democracy affects crucially the conclusions (Table 5). With VAN and VDEM, the relationship between GDP/capita and democracy is positive and significant (at the 99% level), whereas with the other two measures of democracy there is no evidence of a significant relationship between GDP/capita and democracy. Trade openness is a significant predictor of democracy only with PIV and VDEM, education is a significant predictor of democracy only with VAN, and population predicts democracy significantly only with VDEM. The relationship between presidential systems and democracy is significantly positive with PIV (at the 95% level), VAN (at the 99% level), but non-significant with FH and VDEM. Parliamentary systems are positively related to democracy with VAN (at the 99.9% level), whereas with PIV, FH, and VDEM the estimates do not point to the existence of such a relationship. Party fractionalisation has a positive and significant effect on democracy with PIV and VAN, but not with FH and VDEM.

Table 5:

Longitudinal regressions of democracy on structural predictors (1996–2017)

Note: T-values in parentheses; *p < 0.05, **p < 0.01, ***p < 0.001. Constant coefficient measured, but not reported. All models are estimated with OLS-PCSE and include a lagged dependent variable (not reported) on the right-hand side of the regression equation. Independent variables are lagged by 1 year. The years 1996–2017 refer to the dependent variable

The coefficient plots in Fig. 3 provide a visual representation of the point estimates and 95% confidence intervals of selected independent variables in our battery of regression models. These plots illustrate clearly how the lack of interchangeability in the full sample of country-years (1973–2017, first row in Fig. 3) is driven by the weak interchangeability of the measures in the sample of most recent years (1996–2017, third row in Fig. 3). In the antecedent sample of country-years (1973–1995, second row in Fig. 3), on the contrary, the results are almost never different from zero, regardless of the chosen measure.

Fig. 3

Coefficient plots of selected predictors across regression models

Besides summarising the main results of the regressions, coefficient plots are particularly useful to analyse differences in confidence intervals. They allow us to notice immediately that, almost without exceptions, the predicted effects in models with VDEM as the dependent variable have systematically narrower standard errors compared to the other models. Being the cause of less precise predictions can be considered a limitation of any measure, since we can confidently assume that users of measures of democracy prefer precise inferences to imprecise inferences. This finding can be explained by the fact that in general VDEM captures much more variation across countries and over years than the other measures. Hence, VDEM seems to lead to more precise estimates than the other measures and is likely to bring more fruitful results in large-N cross-national studies than VAN, PIV, or FH.

We would expect highly convergent measures of democracy to be similarly related to structural predictors of democracy, but the findings show instead that in general the interchangeability of measures of democracy is weak. Choosing one measure instead of another can affect substantially the conclusions. In fact, in two of our three sets of regression models there is not even a single pair of measures that leads to the same conclusions across all predictors. This is problematic, since most scholars treat highly correlated measures of democracy as interchangeable. The findings show as well that the interchangeability of these measures has weakened over time. While the measures lead to relatively similar conclusions in a longitudinal analysis from 1973 to 1995, the choice of the measure becomes crucial in a longitudinal analysis from 1996 to 2017, where all measures lead to different conclusions.