Introduction

Most scholars agree on the concept of democracy, which has informed empirical social science since Dahl (1971) coined the termed “polyarchy,” combining fair elections under universal suffrage with political liberties and alternative sources of information that enhance the democratic qualities of elections (e.g., Collier and Levitsky 1997; Møller and Skaaning 2010). Despite the popularity of Dahl’s concept of polyarchy, no measure has been available to capture it comprehensively. Based on 40 specific indicators from the Varieties of Democracy (V-Dem) dataset (Coppedge et al. 2018a), we provide nuanced measures of each of the core “institutional guarantees” in Dahl’s (1971, 1989, 1998) conceptualization, across a global set of 182 countries from 1900 to 2017,Footnote 1 and a combined index of polyarchy with point estimates as well as measures of uncertainty.

Table 1 lists the advantages of this polyarchy index over existing graded measures with similar coverage, such as Freedom House (FH), Polity (Marshall et al. 2014), and the UDS (Pemstein et al. 2010). First, by measuring Dahl’s (1998, p. 85) five components of “Elected Officials,” “Free and Fair Elections,” “Freedom of Expression and Alternative Sources of Information,” “Associational Autonomy,” and “Inclusive Citizenship,” this is the first complete measure of Dahl’s polyarchy since Coppedge and Reinicke (1990), which covered only one year. Second, to rate each indicator, V-Dem relies on multiple (as a rule five or more) carefully recruited country experts, the bulk of whom are from the countries they code, thus tapping into a hitherto unparalleled knowledge base about the workings of democracy around the world. Third, the dataset is public and aggregation is fully transparent down to constitutive indicators and coder-level data, which allows for systematic investigation of how these specific aspects affect aggregate scores. Four, we provide systematic measures of uncertainty for all point estimates.

Table 1 Democracy indices compared
Full size table

We also show below that characteristics of V-Dem’s country experts systematically predict neither their ratings nor the differences between the polyarchy index scores and existing measures such as FH and Polity. Furthermore, such differences speak in favor of the face validity of our new measure.

Figure 1 shows how the world looks different with the V-Dem polyarchy index in five of the countries where differences to extant measures are stark. Freedom House and Polity record nearly flawless scores without development over time for both the USA and Costa Rica. Consistent with massive evidence on limited suffrage, intimidation, and persistent fraud at the polls in the past (Keyssar 2000; Lehoucq and Molina 2002), the polyarchy index arguably does a better job in capturing how electoral democracy developed only gradually in both countries, while also picking up a small decline in polyarchy in the USA in recent years. For Colombia, Polity suggests sharp and abrupt improvements during the rule of the Liberal party (1930–1946) while V-Dem displays a relatively constant level due to hampered competition at the polls (Bejarano and Leongómez 2002). Similar results are seen in Malaysia after independence (Case 2004). Freedom House seems increasingly to have caught up with real-world developments by gradually adjusting its scores downward in both cases. South Africa, finally, is a well-known case of a “competitive oligarchy” prior to 1994 (Dahl 1971). Ignoring that the vast majority of the population was disenfranchised, Polity scores the country high on democracy. While capturing the tidal shift around 1994, Freedom House instead seems to underappreciate lingering problems with political intimidation and violence in South Africa (Gibson and Gouws 2003). These cases illustrate that V-Dem is more sensitive to such weaknesses in democracy than extant measures are.

Fig. 1
figure 1

Five examples of key differences with V-Dem polyarchy (90% CIs)

Full size image

Below we contrast polyarchy with other conceptions of democracy and outline V-Dem’s methodology, followed by a section on how we measure the components of polyarchy. The fifth section describes how the polyarchy index is aggregated from its components. We end by probing the validity of the underlying V-Dem data and indices, and our conclusions.

Polyarchy and Minimalist vs. Maximalist Conceptions of Democracy

Dahl (1971, p. 2) famously defined democracy as “a political system one of the characteristics of which is the quality of being completely or almost completely responsive to all its citizens.” He reserved “democracy” for this ideal while proposing a set of empirical requirements for the more realistic standard of “polyarchy.” Originally eight, these institutional guarantees were narrowed down to seven (Dahl 1989) and eventually six (Dahl 1998, p. 85). We have in turn collapsed two of these on theoretical and empirical grounds, leaving the five guarantees presented in Table 2.Footnote 2

Table 2 The five institutional guarantees of polyarchy
Full size table

Dahl’s polyarchy is often juxtaposed with Schumpeter’s (1942, p. 269) “minimal” conception that excludes any reference to political liberties such as freedom of expression and freedom of association. Arguing that “disqualifications on grounds of economic status, religion and sex [are] compatible with democracy,” Schumpeter (1942, pp. 244–5) also excluded extension of the suffrage (Møller and Skaaning 2010, pp. 268–9) and thus defined democracy as only the first, second, and (one aspect of the) third Dahlian institutional guarantees (see Table 2).

Defenses of more minimalist conceptions typically argue that the relationship between democracy and other factors such as civil liberties should be treated as empirical rather than definitional (e.g., Boix et al. 2012, p. 1527; Przeworski et al. 2000, p. 34). Yet, empirical considerations should not override compelling theoretical arguments. Democracy without the inclusion of most of “the people,” or without liberties that make elections meaningful, is an oxymoron. Accordingly, the majority of attempts to measure democracy in the world include suffrage and freedom of expression (e.g., Bollen 1980; Coppedge and Reinicke 1990; Hadenius 1992; Gasiorowski 1996; Mainwaring et al. 2001; Bowman et al. 2005; Coppedge et al. 2008).

Another prominent argument is that a non-electoral dimension such as freedom of expression should not be part of an election-centered measure of democracy. Why not simply refer to “elections, tout court” (e.g., Skaaning et al. 2015, p. 5) and hence reserve the term “electoral democracy” for a more minimalist notion (e.g., Møller and Skaaning 2010, pp. 268–171; Munck 2009, pp. 55–56)? Without denying the need for diminished subtypes for certain analytical purposes (Collier and Levitsky 1997), we maintain that Dahl provides the most comprehensive and most widely accepted theory of what constitutes an electoral democracy. “By this conception,” in Diamond’s (2002, p. 21) words, “democracy requires not only free, fair, and competitive elections, but also the freedoms that make them truly meaningful” (italics added). To avoid the “fallacy of electoralism” (Karl 1986), even election-centered notions of democracy need to take into account these non-electoral aspects (see also O’Donnell 2001).Footnote 3

At the same time, some may find Dahl’s (1971) notion of polyarchy too minimalist since it refers to “democratic procedures, rather than to substantive policies or other outcomes that might be viewed as democratic” (Collier and Levitsky 1997, p. 433). There are also other procedural but still more demanding conceptualizations of democracy than “polyarchy,” such as the liberal tradition stressing strong rule of law and effective checks and balances limiting the use of executive power. However, we strive for a measure that aligns as closely as possible with the conception of democracy agreed upon by most scholars. This happens to be a combination of principles such as Suffrage, Free and Fair Elections, and some basic freedoms such as freedom of association and expression, but not including more demanding features such as checks and balances or the rule of law: i.e., Dahl’s (1971) polyarchy.

Nevertheless, V-Dem data are still useful for those who require more specific measures. Since we measure all components of polyarchy separately, our approach allows one to construct a more limited Schumpeterian measure or to test empirical relationships among specific components, such as how the extent of the suffrage relates to the degree of electoral competitiveness.

V-Dem’s Methodology

Drawing on the large literature on attempts to measure electoral democracy (e.g., Coppedge et al. 2011; Hadenius and Teorell 2005; Munck 2009; Munck and Verkuilen 2002), we argue that no extant measure fulfills all of the five essential criteria: (1) capturing all institutions in Dahl’s concept of polyarchy; (2) providing disaggregated data allowing for analyses of dimensionality and inquiries into what lower-level changes account for the shifts in higher-level indices; (3) covering a global sample of countries across long swathes of time; (4) using transparent data generating processes and aggregation rules; and (5) providing estimates of measurement uncertainty.

Extant measures meet only some of these criteria (for details, see Coppedge et al. 2017). Therefore, a new measure of polyarchy fulfilling all of them is needed. Three features of Varieties of Democracy (V-Dem) data make this possible.Footnote 4 The first feature is radical disaggregation: principles of democracy are translated into detailed questions with well-defined response categories or measurement scales, 40 of which are here used to construct the polyarchy index. We thus measure each polyarchy component (save one: suffrage) with multiple indicators, enhancing reliability and providing a basis for tests of dimensionality.

Second, the bulk of the 40 indicators come from ratings by country experts, mostly academics who are from the 182 countries across the globe that V-Dem covers (see Appendix A).Footnote 5 They are recruited based on academic or credentials as field experts of the country and substantive area they code, their seriousness of purpose, and their impartiality.Footnote 6 The target is that at least five experts rate each indicator for each country and year going back to 1900, thus engaging more than 3000 experts in total.

As a third unique feature, V-Dem employs a custom-designed Bayesian item response theory model to estimate latent country-date traits from the expert ratings (see Pemstein et al. 2018). Patterns of cross-rater (dis)agreement are used to estimate variations in reliability and systematic differences in thresholds between ordinal response categories, in order to adjust estimates of the latent concept. To achieve cross-country and over-time comparability, we also use ratings by over 350 “lateral” coders (who rate multiple countries for a limited time period), over 400 “bridge” coders (who code the full time series for more than one country), and anchoring vignettes (a random selection of which have been answered by roughly 80% of the country experts).

Measuring the Parts: Five Components of Polyarchy

The measurement literature distinguishes between reflective (or “effect”) and formative (“cause” or “constitutive”) indicators. Reflective indicators are “symptoms” of the concept typically estimated by factor analysis, treating each indicator as the result of a common factor and a unique error component trait (Treier and Jackman 2008). Formative indicators reverse the equation, treating indicators as constitutive of the concept (Goertz 2006). While reflective indicators should be correlated and preferably load on a single factor in a factor analysis, there is no such expectation for formative indicators (Bollen 1989).Footnote 7

Both the reflective (Bollen 1990; Treier and Jackman 2008) and the formative (Munck 2009; Goertz 2006) views of democracy measures have their advocates. Summarized in Table 2, we argue that the component of Elected Officials should, on theoretical grounds, be treated as constituted by its indicators, whereas the remaining four indicators are best treated as “effects” of their respective higher-level component. We proceed by discussing them in turn.

Formative Indicators: Elected Officials

This component index measures whether the legislature and the chief executive are elected—the latter either directly through popular elections or indirectly through a popularly elected legislature appointing the executive. This is sometimes referred to as the “effectiveness” (Hadenius 1992, p. 49) or “decisiveness” (O’Donnell 2001, p. 13) of elections. A “popular election” is minimally defined and includes sham elections with limited suffrage and no competition, and “appointment” by the legislature only implies selection and/or approval, not the power to dismiss.Footnote 8 Unlike the other four component indices, this index is based on a set of formative (or constitutive) indicators that define the construct on purely theoretical grounds. The logic behind the construct is schematically portrayed in Fig. 2.

Fig. 2
figure 2

The components of the Elected Officials Index

Full size image

There are six links of appointment/selection of the chief executive to account for: first and second, whether the head of state (a1) and/or head of government (a2) is directly elected (1) or not (0); third, the extent to which the legislature is popularly elected (b), measured as the proportion of legislators who are directly or indirectly elected (if the legislature is unicameral), or if the legislature is bicameral and the upper house is involved in the appointment of the chief executive, a weighted average of the proportion elected for each house;Footnote 9 fourth and fifth, whether the head of state (c1) and/or head of government (c2) is appointed by the legislature (1), or the approval of the legislature is necessary for the appointment of the head of state (1), or not (0); and sixth, whether the head of government is appointed by the head of state (d = 1) or not (d = 0) (see Appendix B on exact question wordings and response categories).

In polities with unified executives (that is, when the head of state is also the head of government, Elgie 1998; Siaroff 2003), the complexity of this conceptual scheme reduces to the links a1 and b*c. Since these are considered perfect substitutes (either a directly elected president or a president elected by a directly elected parliament suffices), the index value is arrived at by taking the maximum value of the two. In dual systems, where there is both a head of state and a separate head of government, the chief executive is determined by comparing the two executives’ power over the appointment and dismissal of cabinet ministers. If the head of state and head of government share equal powers over the appointment and dismissal of cabinet ministers, the index averages across the degree to which they are directly or indirectly elected. The resulting construct is called the elected executive index (v2x_accex).

In the second step, the extent to which the legislature is elected (b) is also independently taken into account in order to penalize presidential systems with unelected legislatures, or legislatures with a large share of presidential appointees, for example. Assuming that the extent to which the chief executive and the legislature are elected work as partial substitutes, we average the two. The resulting index of Elected Officials (called v2x_elecoff) has a bimodal distribution (see Appendix Figure D1). The rare values falling between 1 and 0 are mostly bicameral systems where the upper house is indirectly elected, but also include cases such as Burma/Myanmar that (currently) has a unified executive with a president elected by a parliament in which only 75% of the seats are directly elected.Footnote 10

Reflective Indicators: Bayesian Factor Analysis Indices

The other polyarchy components are arguably better measured by reflective indicators that partially substitute for one another. With the exception of “Inclusive Citizenship,” which we capture by the extension of the suffrage,Footnote 11 we include multiple indicators for each component to allow both for a more comprehensive appraisal of each component, and for assessments of the extent to which they load on a single dimension. To test the latter proposition, we run a Bayesian factor analysis (BFA) model for each of the three components “Free and Fair Elections,” “Associational Autonomy,” and “Freedom of Expression and Alternative Sources of Information.” The results are reported in Appendix Tables A1–3. In each case, there is clear evidence for unidimensionality. We therefore construct a BFA index for each of these components in turn, accompanied by measures of uncertainty.Footnote 12

The first, which Dahl (1998, p. 85) calls “free, fair, and frequent” elections, but which we designate “Clean Elections Index” (v2xel_frefair), is designed to capture the level of integrity of elections (e.g., Schedler 2002; Lehoucq 2003; Birch 2011; Kelley 2013; van Ham and Lindberg 2016). Eight indicators constitute this index, measuring the (a) autonomy and (b) capacity of the election administration body (EMB) to conduct well-run elections; and for each election, the extent of (c) registration irregularities, (d) vote buying, (e) ballot fraud and intentional irregularities, (f) government-induced intimidation of opposition candidates, (g) other types of election violence (not instigated by the government or ruling party), and (h) an overall assessment of whether the election overall was “free and fair.”Footnote 13 The resulting index is bimodally distributed but with a skew towards observations with no elections (see Appendix Figure D4).Footnote 14

The second component captures what Dahl (1998, p. 85) calls “Associational Autonomy” (or freedom of organization) using six indicators. The core of this construct is party-centered: are political parties free to form and operate, and are opposition parties allowed autonomy from the ruling regime? Equally important, and implying that there is an electoral aspect also to this component: are parties free to field candidates in national elections making them truly multiparty? Yet, Associational Autonomy in the political sphere additionally requires that there be no state repression, or barriers to the entry and exit, nor repression of a wider set of civil society organizations providing alternative means for voice and political activity. The resulting index (v2x_frassoc_thick), rescaled to 0–1 through the normal cumulative distribution function (cdf), is bimodally distributed (see Appendix Figure D5).Footnote 15

Finally, Dahl’s concept of polyarchy includes two conspicuously non-electoral aspects: freedom of expression and access to alternative sources of information. Staying true to his widely accepted concept thus necessitates employing a core set of indicators capturing overall media freedom (Behmer 2009), such as active state censorship of print/broadcast media, media self-censorship, and harassment of journalists. In addition, it requires freedom of discussion in society at large, for both men and women (Skaaning 2009), and four indicators of media content to capture Dahl’s (1971, 1989, 1998) “alternative sources of information”: whether the media is biased against opposition parties and candidates, whether major print and broadcast outlets routinely criticize the government, and whether they represent a wide range of political perspectives, as well as general repression of cultural and academic expressions of political dissent. As argued above, one could question whether these really belong to a mostly electoral conception of democracy. We contend that Dahl (1989, 1998) himself provided the most viable answer to that question through his treatment of the notion of “enlightened understanding.” In essence, elections may be free and fair, and suffrage universal, yet without these non-electoral institutional guarantees in place, elections would still be meaningless if voters do not make an informed choice based on at least some minimal possibilities for collective deliberation. We therefore include both these guarantees using nine indicators but collapse them into one index called Freedom of Expression and Alternative Sources of Information (v2x_freexp_altinf) since they are very strongly correlated (results in Appendix Table A3). This is again converted to a probability (0–1) score by using the normal cdf, resulting in the distribution in Appendix Figure D7.

Measuring the Whole: Aggregating the Components

When determining how to aggregate the components into the overall polyarchy measure, the most important consideration is whether the five components should be seen as (partially) substitutable aspects of polyarchy, or if they form a set of necessary conditions. The literature is unfortunately divided on this matter. On one hand, there is a strong foundation for treating the components as necessary conditions (see Przeworski et al. 2000; Munck 2009; Boix et al. 2012). This argument holds that the degree of suffrage is not relevant if there is no Associational Autonomy, if the election results are completely fabricated, or if the executive is not elected. Similarly, by this logic, the freedom and fairness of elections should not count if only a tiny fraction of the population is enfranchised, and so on. What we call the Multiplicative Polyarchy Index (MPI) is following this “necessary conditions” logic:Footnote 16

$$ MPI= Elected\ Officials\ast Clean\ Elections\ast Associational\ Autonomy\ast Suffrage\ast Freedom\ of\ Expression\ and\ Alternative\ Source\ of\ Information $$
(1)

A low score on any of the component indices thus suppresses the value of the overall index. As a result, the distribution is heavily skewed towards zero (see the upper left quadrant of Fig. 3). The measures of uncertainty in the lower-level component indices are propagated into this component index by multiplying the standard errors from the BFA posteriors of the Clean Elections, the Associational Autonomy, and the Freedom of Expression and Alternative Source of Information indices (since both Elected Officials and Suffrage are assumed to be measured without error).

Fig. 3
figure 3

Aggregating to polyarchy

Full size image

Nonetheless, there is also a second well-established strand in the literature on how to measure electoral democracy going back to, for example, Bollen (1980), Coppedge and Reinicke (1990), and Hadenius (1992). According to this argument, the polyarchy components are correlated and, to some extent, interchangeable measures of the same underlying construct. The aggregation rule should thus be additive rather than multiplicative. This logic seems to have its strongest theoretical rationale when it comes to the components based on “freedoms” of different sorts, particularly with respect to O’Donnell and Schmitter’s (1986) concept of “liberalization”—the phase in a transition to democracy when the first opening of the authoritarian regime occurs (such as a lifting of media censure and wider acceptance for expressions of popular discontent), before the first “founding election.” If the extent to which such “liberalizations” count is made conditional on the electoral side of the equation—as implied by the multiplicative logic—they do not register in the data while arguably happening on the ground before elections are held. The additive or averaging logic, however, allows such openings to matter independently.

Yet, as Dahl (1971) argued, and as recently demonstrated by Coppedge et al. (2008) and Miller (2013), there are other paths towards such “partial polyarchy.” Another prominent route is the introduction of executive elections with universal suffrage, but with severe electoral manipulation, little or no competition at the polls, and severe repression of the freedom of expression. If we generalize the additive logic to the full set of components, these two different paths would weigh approximately equally in the resulting index. While recognizing the value of an aggregation rule that also lets the polyarchy components influence the overall score independently of one another, we therefore favor a hybrid approach: we let the two components that can achieve high scores based on the fulfillment of formal-institutional criteria (Elected Officials and Suffrage) together weigh half as much as the other components that enjoy a stronger independent standing in terms of respect for democratic rights (Clean Elections, Associational Autonomy, and Freedom of Expression and Alternative Sources of Information). The Additive Polyarchy Index (API) is thus computed as follows:Footnote 17

$$ API=\left[\left( Elected\ Officials+ Suffrage\right)+2\ast \left( Clean\ Elections+ Associational\ Autonomy+ Freedom\ of\ Expression\ and\ Alternative\ Source\ of\ Information\right)\right]/8=.125\ast Elected\ Officials+.125\ast Suffrage+.25\ast Associational\ Autonomy+.25\ast Clean\ Elections+.25\ast Freedom\ of\ Expression\ and\ Alternative\ Source\ of\ Information $$
(2)

Measurement uncertainty is here taken into account by averaging the standard errors from the BFA estimates of Clean Elections, Associational Autonomy, and Freedom of Expression and Alternative Sources of Information. The resulting index is mostly distributed bimodally, as shown in the upper right quadrant of Fig. 3. The lower left quadrant of Fig. 3 contrasts the two aggregation rules, demonstrating that they discriminate at two different ends of the underlying scale. Thus, the additive index mostly differentiates among different degrees of democracy at the lower end of the multiplicative scale. Even when the multiplicative index is zero, the additive index can achieve as high a score as .77. Conversely, the multiplicative index mostly discriminates among countries already achieving high values on the additive scale. Thus, the multiplicative scale varies from .16 to .90 when the additive scale is above .80. This has potentially significant implications for the results of many empirical analyses.

Since both the multiplicative and the additive logic have support in the literature, and since they evidently have the virtue of discriminating at different ends of the spectrum of a foundational concept in political science that one would want to measure in full, we argue that the average of the two is the preferred solution to the aggregation problem. The V-Dem electoral democracy (polyarchy) index is thus constructed by averaging (1) and (2), or more precisely:Footnote 18

$$ \mathrm{Polyarchy}=.5\ \mathrm{MPI}+0.5\ \mathrm{API}=.5\ast Elected\ Officials\ast Clean\ Elections\ast Associational\ Autonomy\ast Suffrage\ast Freedom\ of\ Expression+.0625\ast Elected\ Officials+.125\ast Clean\ Elections+.125\ast Associational\ Autonomy+.0625\ast Suffrage+.125\ast Freedom\ of\ Expression $$
(3)

As shown in the lower right quadrant of Fig. 3, this index is still positively skewed, but far less so than the multiplicative version. In Fig. 4, we display the ranking of countries in our sample in the last year for which we currently have complete country data, 2017, along with 90% confidence intervals. In this particular year, Saudi Arabia, Eritrea, Qatar, China, North Korea, and Laos score at the very bottom, whereas Costa Rica, Denmark, Switzerland, France, Sweden, Norway, and Estonia are ranked highest. A large group of mostly Western established democracies, but also including countries such as Chile, Uruguay, Japan, Jamaica, Mauritius, and Taiwan, are indistinguishable at the top in terms of polyarchy scores when taking the confidence intervals into account. Measurement error tends to be largest in the middle, as expected, since it should be easier to pin down the level of polyarchy in advanced democracies and in closed authoritarian regimes than in the “muddled middle.” This is much more intuitive than previous findings that measurement uncertainty is largest at the extremes (Treier and Jackman 2008).

Fig. 4
figure 4

Polyarchy in 2017 (90% CIs)

Full size image

Validating the Polyarchy Index

Following McMann et al. (2016), there are two fundamentally different ways in which we could provide positive evidence of validity. The first is to compare the ratings of different country experts for the same indicators, countries and years; the second is to compare these coder-level ratings, as well as the aggregated index at the country-year level, to other similar measures of electoral democracy from other datasets. Under the general heading of “convergent validity” (Campbell and Fiske 1959), agreement among coders or among datasets are both considered as evidence of measurement validity. Moreover, following Donchev and Ujhelyi (2014), understanding the sources of disagreement can provide an additional tool for assessing validity. In what follows, we pursue both strategies of validation.

Comparing Coders

Assuming that V-Dem experts exercise independent judgment when translating their perceptions of the world into numerical ratings (Schedler 2012), coder agreement could be interpreted as a sign of validity (Steenbergen and Marks 2007, p. 351). Table 3 decomposes the variance in the 23 expert-coded indicators (out of the 40 in total) underlying the polyarchy index: Clean Elections (8), Associational Autonomy (6), and Freedom of Expression and Alternative Sources of Information (9).Footnote 19 Results are pooled across indicators for each component, controlling for country- and year-fixed effects. With random effects measured on the same ordinal 0–4 scale as the original indicators to ease interpretation, the standard deviations are thus only about a half point on a four-point scale suggesting very modest levels of disagreement among the country experts. One should also recall that a decent share of this “raw” coder disagreement is known to be a reflection of varying coder thresholds for mapping perceptions onto the ordinal coding categories, and that the measurement model corrects a substantial share of this differential item functioning when aggregating the raw scores to the country-year level (Pemstein et al. 2018).

Table 3 Variance decomposition of coder disagreement
Full size table

Granted that country experts disagree at times, are there predictable patterns in that incongruity? Arguably, we should expect coders to disagree more, the more difficult the coding task (Steenbergen and Marks 2007; Coma and van Ham 2015). In V-Dem data, we may distinguish at least three sources of difficulty (McMann et al. 2016). First, all else equal, judging indicators of electoral democracy should be more difficult further back in time, when experts cannot rely as much on their academic and lived experience. Second, difficulty should increase when there is little independent information from which to form one’s judgment. Third, extreme conditions should be more easily identifiable than the “muddled middle”; hence, coders should be more likely to disagree at intermediate levels of whatever indicator they are assessing. This third expectation may work counter to the first to the extent that the situation in many countries was quite undemocratic aspects in the early years of the twentieth century.

In Table 4, we test these three propositions by regressing the country-year standard deviation in coder ratings, pooled across the same 23 expert-coded indicators as in Table 2, on year of observation, an independent measure of media access,Footnote 20 the level of the aggregate coder rating (a product of the measurement model), and that level squared. We present results by component, controlling for indicator-fixed effects (not displayed in the table). We also control for the number of coders, which may in itself be a systematic driver of the amount of disagreement.Footnote 21 All three expectations are borne out by evidence. Coders disagree more when coding electoral democracy indicators further back in time, when there is less available media information, and at intermediate levels of the indicators they are assessing. There are only two exceptions. First, for Associational Autonomy, the largest degree of coder disagreement is actually at the lowest levels; second, the estimate of observation year for the clean elections indicators is insignificant (implying that election quality is equally difficult to assess across time when holding the amount of available information constant). We conclude that by and large, the level of coder disagreement varies in a meaningful and predictable way, lending further support to the validity of V-Dem data.

Table 4 Predicting coder disagreement
Full size table

As a final test comparing coder estimates, we assess whether coder characteristics from the V-Dem post-survey questionnaire systematically affect ratings following Dahlström et al. (2012). We control for gender (female = 1, male = 0); age (in years, also including the squared polynomial); level of education (PhD = 1, 0 = less than PhD); employment (1 = government, 0 = other); whether the expert was born in or resides in the country he or she is coding (1 = yes, 0 = no); and whether the expert is of “Western” origin or not. Additionally, we control for three potential sources of ideological bias: support for free market economy as a proxy for left-right leaning, support for two different normative understandings of democracy. The first we call the “conventional understanding,” which entails support for electoral and liberal democracy. The second we call the “alternative understanding,” which entails support for participatory, deliberative, or egalitarian types of democracy.Footnote 22Appendix F provides the exact question wording for all the above. In order to fix comparisons across coders to the same countries and years, we also control for country- and year-fixed effects (results omitted).

The results in models 1, 3, and 5 of Table 5 are comforting overall. With two exceptions, characteristics are not significantly related to how V-Dem experts rate electoral democracy indicators. The exceptions are (a) the tendency among coders born in the country to provide a more positive assessment of election quality than non-native experts and (b) those supportive of “alternative” understandings of democracy to provide lower ratings on freedom of expression than “conventionalists.” These are interesting patterns worthy of further inquiry, but they are the exceptions to the rule: experts tend to provide similar assessments of election quality, freedom of association or of expression, independent of who they are.

Table 5 Predicting coder ratings
Full size table

Models 2, 4, and 6 test for another type of coder bias that Bollen and Paxton (2000, p. 72) label “situational closeness.” This is the idea that “judges will be influenced by how situationally and personally similar a country is to them. “Accordingly, strong believers in the free market would have a specific tendency to rate countries with free markets better. Similarly, situational closeness bias implies that believers in other conceptions of democracy rank countries more proximate to their ideals more favorably also on the electoral indicators. Finally, this kind of bias could manifest in such a way that even if Western coders displayed no systematic bias for their countries, bias might be directed towards non-Western countries. We test this by controlling for interaction effects between particular coder and country characteristics.Footnote 23 None of these interaction effects are, however, significant at conventional levels. The dominant pattern is thus that situational closeness does not introduce bias in the V-Dem experts’ ratings of electoral democracy.

Comparing Measures

A second set of validation exercises focus on comparisons between our measure of electoral democracy and extant alternatives. Figure 5 displays the bivariate descriptive pattern at country-year level, comparing the V-Dem polyarchy index to the Polity and Freedom House ratings, as well as the “Unified Democracy Scores” (UDS). The latter is based on just about every pertinent measure of electoral democracy (except V-Dem) covering multiple countries and years (Pemstein et al. 2010). Convergence is the overall pattern. The pairwise correlations range from .85 for Polity to .94 for the UDS ratings. The red smoothed lowess-line of best fit also indicates a consistent pattern of monotonically increasing levels of polyarchy, the higher the extant measure of electoral democracy. Without treating any of the particular measures as gold standards, in the convergent validity sense, these different measures validate one another.

Fig. 5
figure 5

Comparing polyarchy with Alternative Electoral Democracy Indices. Note: The black dotted lines are smoothed lowess regression lines of best fit

Full size image

Nonetheless, the polyarchy index regularly produces scores that differ from extant measures. Some pertinent examples of this were highlighted already in Fig. 1, where we argued that the V-Dem measure outperforms extant measures in terms of face validity. In Figs. 6 and 7, we portray the country differences more systematically by plotting the average polyarchy scores against the average Polity and FH scores, taking into account every country year that exists in both respective datasets.Footnote 24 The diagonal indicates where there is no difference between the measures, implying that V-Dem polyarchy rates electoral democracy higher than the alternative measures for cases above the line, but lower for the ones below the line. There are a few cases above the diagonal (the top five are highlighted in red). Three cases appear on this list for both Polity and Freedom House: the German Democratic Republic (GDR), Turkmenistan, and Uzbekistan. The most important reason that V-Dem tends to score those countries higher than both Polity and Freedom House is that Elected Officials and Suffrage, which both score high in these cases, are allowed to exert an independent influence on the polyarchy scores.

Fig. 6
figure 6

Comparing average polyarchy with Polity scores

Full size image
Fig. 7
figure 7

Comparing average polyarchy with FH scores

Full size image

The bulk of cases, however, appear below the line (the average difference with Polity is − .13, with FH − .06), meaning that the polyarchy index is a more demanding standard on average than the alternative measures. Several of the most diverging cases for Polity are presented and discussed already in Fig. 1. (Colombia ranks 12th in terms of how far it falls below the diagonal, so it is not highlighted in Fig. 6.) But for Freedom House, there is an additional conspicuous pattern: several of the countries where FH is significantly larger than polyarchy are relatively small states, such as Barbados, the Dominican Republic, Montenegro, Papua New Guinea, and the Solomon Islands. What drags down these countries in particular in the polyarchy index is the low quality of their elections as compared to their FH ratings.

In sum, the polyarchy index is positively correlated with extant measures of democracy, but we also find discrepancies where we believe polyarchy, by incorporating all of Dahl’s five components and by relying on more fine-grained case expertise, provides a more nuanced and accurate picture of the state of electoral democracy.Footnote 25

Conclusion

This paper presents a new index of polyarchy that has several advantages over extant measures. The V-Dem polyarchy index is firmly anchored in the most established canon of positive democratic theory. It is based on a unique knowledge base and data-generating process that addresses thorny issues regarding cross-country and cross-rater comparability while also providing measures of uncertainty. Moreover, its five components with their respective indices, and their constituent 40 indicators, make it possible to drill down into what exactly drives changes in scores of country-specific trajectories. This transparency and detail enables researchers to pursue novel research agendas such as the relationship of economic development, democratic peace, human development, corruption, or state capacity to disaggregated components of democracy. The index and its construction also allow end users to choose between two variants representing the two main traditions in terms of viewing Dahl’s institutions as either necessary or partly compensatory conditions of democracy, and to work with thinner conceptions of electoral democracy when they are more appropriate for their research questions.

Finally, several tests of validity of the new polyarchy index strengthen our belief that it is by and large not driven by biases stemming from coder characteristics or ideological predilections. There is some amount of coder agreement for individual country-year indicators but we have demonstrated that deviations from that pattern can be meaningfully understood in predictable ways that do not necessarily undermine the validity of the index. We also find strong correlations with other existing measures of electoral democracy, but also decisive differences where we believe the evidence supports that the polyarchy index has higher face validity.

We hope that this new resource will prove useful to scholars in bringing forward new and more robust knowledge on the causes and consequences of democratization. While this new measure will not solve all issues with democracy research, we hope it will alleviate some of them and open up new research agendas.