Skip to main content
SpringerLink
Log in

assessing the quality of quality measures of democracy: a theoretical framework and its empirical application

  • Research
  • Published:
European Political Science Aims and scope Submit manuscript

Abstract

Over recent decades, comparative political scientists have developed new measures at a rate of knots that evaluate the quality of democratic regimes. These indices have been broadly applied to assess the quality of democracy cross-nationally and to test the generalisability of theories regarding its causes and effects. However, the validity of these inferences is jeopardised by the fact that the quality of democracy is an abstract and contested concept. In order to address this eventuality, researchers constructing indices measuring the quality of democracy as well as researchers applying these indices should critically examine the quality of the indices. Owing to the absence of a standardised framework that is both suitable for the evaluation of contested concepts and that includes explicit coding rules so as to be directly applicable, this article seeks to fill this gap. The application of our framework is demonstrated by an evaluation of the Sustainable Governance Indicators, the Global Democracy Ranking and the Democracy Barometer. As indicated by our evaluation, the framework is a practical tool that helps to assess the conceptual foundation, validity, reliability and replicability of indices. In addition, it can be used to study the quality of indices in a comparable manner.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. We define bias as a ‘systematic and culpable error’ (Hammersley and Gomm, 1997: 4.13). While bias can affect the quality of research at all stages of the research design (for an overview see Indrayan, n.d.), given the task at hand, we focus on how to prevent biases arising from the perspective of the researcher of the concept as well as the measurement of the concept (Gujarati, 2004: 65–67; Moses, 2011: 798–801).

  2. Lauth has developed an index that serves to measure the quality of political regimes, the so-called ‘Combined Index of Democracy’ (CID). Given the intention of this index, it goes beyond the scope of this article (Lauth and Kauff, 2012).

  3. For a detailed explanation of the relationship between these dimensions see Morlino (2011: 197–211) and Diamond and Morlino (2004: 7–29).

  4. Since some criteria are dichotomous by definition, they had to be scaled accordingly. Consequently, the dichotomous criteria receive a larger weight in the evaluation. This slightly biases the positive or negative tendency of the quality assessment (Müller and Pickel, 2007: 518). However, dichotomising all of the criteria would lead to an overly harsh evaluation of those criteria where the indices fulfil the standards to some extent.

  5. We must concede that all evaluations are influenced by individual presuppositions, knowledge and limitations. We suggest to make a virtue out of necessity, plead for transparency and propose to leave it to the scientific community to make a judgment.

  6. Gerring (1999: 370–84; 2001: 50–60) includes additional criteria that we do not take into account because they pertain to the usefulness of the concept as such and how it is received by the scientific community. In our case, we take this as given.

  7. Munck and Verkuilen (2002: 18) caution not to take the results of reliability tests as indicative of the validity of a measure since indices can be highly reliable but invalid because of similar biases among the coders.

  8. For detailed information on the SGI see Schraad-Tischler and Seelkopf (2014).

  9. All members of the research team are listed online (Democracy Barometer, 2014a).

  10. For detailed information on the DB see Merkel et al (2014a) and Merkel et al (2014b).

  11. The country sample includes all countries that were considered ‘free’ or ‘partly free’ according to Freedom House throughout the respective 2-year period of analysis.

  12. For detailed information on the GDR see Campbell et al (2012) and Campbell and Pölzlbauer (2008).

References

  • Adcock, R. and Collier, D. (2001) ‘Measurement validity: A shared standard for qualitative and quantitative research’, The American Political Science Review 95 (3): 529–546.

    Article  Google Scholar 

  • Altman, D. and Pérez-Liñán, A. (2002) ‘Assessing the quality of democracy: Freedom, competitiveness and participation in eighteen Latin American countries’, Democratization 9 (2): 85–100.

    Article  Google Scholar 

  • Babbie, E.R. (2004) The Practice of Social Research, Southbank: Wadsworth Thomson Learning.

    Google Scholar 

  • Bowman, K., Lehoucq, F. and Mahoney, J. (2005) ‘Measuring political democracy: Case expertise, data adequacy, and Central America’, Comparative Political Studies 38 (8): 939–970.

    Article  Google Scholar 

  • Bühlmann, M., Merkel, W., Müller, L., Giebler, H. and Weβels, B. (2012a) ‘Demokratiebarometer: ein neues Instrument zur Messung von Demokratiequalität’, Zeitschrift für Vergleichende Politikwissenschaft 6 (1): 115–159.

    Article  Google Scholar 

  • Bühlmann, M., Merkel, W., Müller, L. and Weβels, B. (2012b) ‘The democracy barometer: A new instrument to measure the quality of democracy and its potential for comparative research’, European Political Science 11 (4): 519–536.

    Article  Google Scholar 

  • Campbell, D.F. (2008) ‘The basic concept for the democracy ranking of the quality of democracy’, available at: http://www.democracyranking.org/downloads/basic_concept_democracy_ranking_2008_A4.pdf, accessed 25 September 2014.

  • Campbell, D.F., Barth, T.D., Pölzlbauer, P. and Pölzlbauer, G. (2012) Democracy Ranking: The Quality of Democracy in the World, Norderstedt: Books on Demand.

    Google Scholar 

  • Campbell, D.F. and Pölzlbauer, G. (2008) ‘The democracy ranking 2008 of the quality of democracy: Method and ranking outcome’, available at: http://www.democracyranking.org/downloads/method_ranking_outcome_2008_A4.pdf, accessed 25 September 2014.

  • Campbell, D.F., Pölzlbauer, P., Barth, T.D. and Pölzlbauer, G. (2011) ‘Das “democracy ranking 2011 of the quality of democracy”, Erstveröffentlichung, Konzept und Kontext’, available at: http://democracyranking.org/wordpress/ranking/2011/data/Democracy_Ranking_Concept_Earlyrelease_German_2011.pdf, accessed 29 September 2014.

  • Campbell, D.F. and Sükösd, M. (2002) ‘Feasibility study for a quality ranking of democracies’, available at: http://www.democracyranking.org/downloads/feasibility_study-a4-e-01.pdf, accessed 25 September 2014.

  • Campbell, D.F. and Sükösd, M. (2003) ‘Global quality ranking of democracies: Pilot ranking 2000’, available at: http://democracyranking.org/wordpress/ranking/2000/data/folder_a4-e-03.pdf, accessed 25 September 2014.

  • Czada, R. (2010) ‘Good Governance als Leitkonzept für Regierungshandeln: Grundlagen, Anwendungen, Kritik’, in A. Benz and N. Dose (eds.) Governance. Regieren in komplexen Regelsystemen, Wiesbaden: VS Verlag für Sozialwissenschaften, pp. 201–224.

    Chapter  Google Scholar 

  • Dahl, R.A. (1970) A Preface to Democratic Theory, Chicago: University of Chicago Press.

    Google Scholar 

  • Democracy Barometer. (2014a) ‘About us’, available at: http://www.democracybarometer.org/about_en.html, accessed 29 September 2014.

  • Democracy Barometer. (2014b) ‘Concept’, available at: http://democracybarometer.org/concept_en.html, accessed 29 September 2014.

  • Diamond, L.J. and Morlino, L. (2004) ‘The Quality of Democracy’, CDDRL Working Papers, Stanford, 20: 1–37.

  • Freedom House. (2014) ‘Freedom in the world’, available at: http://www.freedomhouse.org/report-types/freedom-world, accessed 29 September 2014.

  • Fuchs, D. and Roller, E. (2009) ‘Die Konzeptualisierung der Qualität von Demokratie: eine kritische Diskussion aktueller Ansätze’, in A. Brodocz, M. Llanque and G.S. Schaal (eds.) Bedrohungen der Demokratie, Wiesbaden: VS Verlag für Sozialwissenschaften, pp. 77–96.

    Chapter  Google Scholar 

  • Gastil, R.D. (1988) Freedom in the World: Political Rights and Civil Liberties 1987–1988, New York: Freedom House.

    Google Scholar 

  • Gerring, J. (1999) ‘What makes a concept good? A criterial framework for understanding concept formation in the social sciences’, Polity 31 (3): 357–393.

    Article  Google Scholar 

  • Gerring, J. (2001) Social Science Methodology: A Criterial Framework, Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Goertz, G. (2008) ‘Concepts, Theories, and Numbers: A Checklist for Constructing, Evaluating, and Using Concepts or Quantitative Measures’, in J.M. Box-Steffensmeier, H.E. Brady and D. Collier (eds.) The Oxford Handbook of Political Methodology, Oxford, New York: Oxford University Press, pp. 97–118.

    Google Scholar 

  • Gujarati, D.N. (2004) ‘Bias’, in M.S. Lewis-Beck, A. Bryman and T.F. Liao (eds.) The Sage Encyclopedia of Social Science Research Methods, Thousand Oaks, CA: Sage, pp. 65–67.

    Google Scholar 

  • Hammersley, M. and Gomm, R. (1997) ‘Bias in social research’, Sociological Research Online 2(1). 4–13.

  • Hopkin, J. (2010) ‘The Comparative Method’, in D. Marsh and G. Stoker (eds.) Theory and Methods in Political Science, Basingstoke: Palgrave Macmillan, pp. 285–307.

    Google Scholar 

  • Indrayan, A. (n.d.) ‘Varieties of bias to guard against’, available at: http://www.medicalbiostatistics.com/Types%20of%20bias.pdf, accessed 24 September 2014.

  • Jäckle, S., Wagschal, U. and Bauschke, R. (2012) ‘Das Demokratiebarometer: “Basically theory driven”?’ Zeitschrift für Vergleichende Politikwissenschaft 6 (1): 99–125.

    Article  Google Scholar 

  • Jäckle, S., Wagschal, U. and Bauschke, R. (2013) ‘Allein die Masse macht’s nicht: Antwort auf die Replik von Merkel et al. zu unserer Kritik am Demokratiebarometer’, Zeitschrift für Vergleichende Politikwissenschaft 7 (2): 143–153.

    Article  Google Scholar 

  • Jackman, S. (2008) ‘Measurement’, in J.M. Box-Steffensmeier, H.E. Brady and D. Collier (eds.) The Oxford Handbook of Political Methodology, Oxford, New York: Oxford University Press, pp. 119–151.

    Google Scholar 

  • Juni, S. (2007) ‘Reliability Theory’, in N.J. Salkind and K. Rasmussen (eds.) Encyclopedia of Measurement and Statistics, Thousand Oaks, CA: Sage, pp. 834–835.

    Google Scholar 

  • Kaina, V. (2008) ‘Die Messbarkeit von Demokratiequalität als ungelöstes Theorieproblem’, Politische Vierteljahresschrift 49 (3): 518–524.

    Article  Google Scholar 

  • King, G., Keohane, R.O. and Verba, S. (1994) Designing Social Inquiry: Scientific Inference in Qualitative Research, Princeton, NJ: Princeton University Press.

    Google Scholar 

  • Kneuer, M. (2011) ‘Deficits in Democratic Quality? The Effects of Party-System Institutionalisation on the Quality of Democracy in Central Eastern Europe’, in G. Erdmann and M. Kneuer (eds.) Regression of Democracy?, Wiesbaden: VS Verlag für Sozialwissenschaften, pp. 133–171.

    Chapter  Google Scholar 

  • Landman, T., Beetham, D., Carvalho, E. and Weir, S. (2008) Assessing the Quality of Democracy: An Overview of the International IDEA Framework, Stockholm: International Institute for Democracy and Electoral Assistance.

    Google Scholar 

  • Lauth, H.-J. (2004) Demokratie und Demokratiemessung: eine konzeptionelle Grundlegung für den interkulturellen Vergleich, Wiesbaden: VS Verlag für Sozialwissenschaften.

    Book  Google Scholar 

  • Lauth, H.-J. (2011) ‘Qualitative Ansätze der Demokratiemessung’, Zeitschrift für Staats- und Europawissenschaften 9 (1): 49–77.

    Article  Google Scholar 

  • Lauth, H.-J. (2015) ‘The matrix of democracy: A three-dimensional approach to measuring the quality of democracy and regime transformations’, Würzburger Arbeitspapiere zur Politikwissenschaft und Sozialforschung (WAPS) 6: 1–30.

    Google Scholar 

  • Lauth, H.-J. and Kauff, O. (2012) ‘Demokratiemessung: der KID als aggregiertes Maß für die komparative Forschung. Empirische Befunde der Regimeentwicklung von 1996 bis 2010’, Würzburger Arbeitspapiere zur Politikwissenschaft und Sozialforschung (WAPS) 2: 1–55.

    Google Scholar 

  • Levine, D.H. and Molina, J.E. (eds.) (2011) ‘Measuring the Quality of Democracy’, in The Quality of Democracy in Latin America, Boulder, CO: Lynne Rienner Publishers, pp. 21–37.

    Google Scholar 

  • Marshall, M.G., Gurr, T.R. and Jaggers, K. (2014) ‘Polity IV project: Political regime characteristics and transitions, 1800–2013. Dataset Users’ Manual’, available at: http://www.systemicpeace.org/inscr/p4manualv2013.pdf, accessed 29 September 2014.

  • Merkel, W., Bochsler, D., Bousbah, K., Bühlmann, M., Giebler, H., Hänni, M., Heyne, L., Müller, L., Ruth, S. and Wessels, B. (2014a) ‘Democracy barometer: Methodology’. Version 4, available at: http://democracybarometer.org/Data/Methodical%20Explanatory%201990-2012.pdf, accessed 29 September 2014.

  • Merkel, W., Bochsler, D., Bousbah, K., Bühlmann, M., Giebler, H., Hänni, M., Heyne, L., Müller, L., Ruth, S. and Wessels, B. (2014b) ‘Democracy barometer. Codebook’. Version 4.1, available at: http://www.democracybarometer.org/Data/Codebook_all%20countries_1990-2012_v0914.pdf, accessed 29 September 2014.

  • Merkel, W., Tanneberg, D. and Bühlmann, M. (2013) ‘“Den Daumen senken”: Hochmut und Kritik’, Zeitschrift für Vergleichende Politikwissenschaft 7 (1): 75–84.

    Article  Google Scholar 

  • Morlino, L. (2004) ‘What is a “good” democracy?’ Democratization 11 (5): 10–32.

    Article  Google Scholar 

  • Morlino, L. (2011) Changes for Democracy: Actors, Structures, Processes, Oxford: Oxford University Press.

    Book  Google Scholar 

  • Moses, J.W. (2011) ‘Epistemological and Methodological Foundations’, in B. Badie, D. Berg-Schlosser and L. Morlino (eds.) International Encyclopedia of Political Science, Los Angeles: Sage, pp. 791–802.

    Google Scholar 

  • Mueller, C.W. (2004) ‘Conceptualization, Operationalization, and Measurement’, in M.S. Lewis-Beck, A. Bryman and T.F. Liao (eds.) The Sage Encyclopedia of Social Science Research Methods, Thousand Oaks, CA: Sage, pp. 161–165.

    Google Scholar 

  • Müller, T. and Pickel, S. (2007) ‘Wie lässt sich Demokratie am besten messen? Zur Konzeptqualität von Demokratie-Indizes’, Politische Vierteljahresschrift 48 (3): 511–539.

    Article  Google Scholar 

  • Müller, T. and Pickel, S. (2008) ‘Antwort auf die Replik von Marc Bühlmann, Wolfgang Merkel, Lisa Müller und Bernhard Weßels zum Forumsbeitrag von Thomas Müller und Susanne Pickel’, Politische Vierteljahresschrift 49 (1): 123–126.

    Article  Google Scholar 

  • Munck, G.L. (2012) ‘Conceptualizing the quality of democracy: The framing of a new agenda for comparative politics’. DISC Working Paper 2012/23, available at: https://disc.ceu.hu/sites/default/files/field_attachment/page/node-3320/discwp23.pdf, accessed 29 September 2014.

  • Munck, G.L. (2014) ‘What is democracy? A reconceptualization of the quality of democracy’, Democratization 47: 1–26.

    Google Scholar 

  • Munck, G.L. and Verkuilen, J. (2002) ‘Conceptualizing and measuring democracy. Evaluating alternative indices’, Comparative Political Studies 35 (1): 5–34.

    Article  Google Scholar 

  • Nuscheler, F. (2009) ‘Good governance. Ein universelles Leitbild von Staatlichkeit und Entwicklung?’ INEF Report 96/2009, available at http://inef.uni-due.de/page/documents/Report96.pdf, accessed 29 September 2014.

  • O’Donnell, G. (2004) ‘Human Development, Human Rights, and Democracy’, in G. O’Donnell, J. Vargas Cullel and O.M. Iazzetta (eds.) The Quality of Democracy: Theory and Applications, Notre Dame: University of Notre Dame Press, pp. 9–92.

    Google Scholar 

  • OECD. (2008) Handbook on Constructing Composite Indicators: Methodology and User Guide, Paris: OECD.

  • Przeworski, A. and Teune, H. (1970) The Logic of Comparative Social Inquiry, Malabar: Robert E. Krieger Publishing.

    Google Scholar 

  • Ringen, S. (2007) What Democracy is For: On Freedom and Moral Government, Princeton: Princeton University Press.

    Google Scholar 

  • Sartori, G. (ed.) (1984) ‘Guidelines for Concept Analysis’, in Social Science Concepts: A Systematic Analysis, Beverly Hills: Sage, pp. 15–85.

    Google Scholar 

  • Schedler, A. (2012a) ‘The measurer’s dilemma: Coordination failures in cross-national political data collection’, Comparative Political Studies 45 (2): 237–266.

    Article  Google Scholar 

  • Schedler, A. (2012b) ‘Judgment and measurement in political science’, Perspectives on Politics 10 (1): 21–36.

    Article  Google Scholar 

  • Schmidt, M.G. (2010) Demokratietheorien: eine Einführung, Wiesbaden: VS Verlag für Sozialwissenschaften.

    Book  Google Scholar 

  • Schraad-Tischler, D. and Seelkopf, L. (2014) ‘Concept and Methodology: Sustainable Governance Indicators 2014’, available at: http://www.sgi-network.org/docs/2014/basics/SGI2014_Concept_and_Methodology.pdf, accessed 25 September 2014.

  • Seawright, J. and Collier, D. (2014) ‘Rival strategies of validation: Tools for evaluating measures of democracy’, Comparative Political Studies 47 (1): 111–138.

    Article  Google Scholar 

  • SGI. (2014a) ‘SGI 2014 codebook’, available at: http://www.sgi-network.org/docs/2014/basics/SGI2014_Codebook.pdf, accessed 25 September 2014.

  • SGI. (2014b) ‘Policy performance and governance capacities in the OECD and EU’, available at: http://www.sgi-network.org/docs/2014/basics/SGI2014_Overview.pdf, accessed 25 September 2014.

  • SGI. (2014c) ‘Quality of democracy’, available at: http://www.sgi-network.org/2014/Democracy/Quality_of_Democracy, accessed 29 September 2014.

  • Stevens, S.S. (1946) ‘On the theory of scales of measurement’, Science 103 (2684): 677–680.

    Article  Google Scholar 

  • Stoiber, M. (2011) Die Qualität von Demokratien im Vergleich: zur Bedeutung des Kontextes in der empirisch vergleichenden Demokratietheorie, Baden-Baden: Nomos.

    Book  Google Scholar 

  • van de Vijver, F.J.R. (1998) ‘Towards a Theory of Bias and Equivalence’, in J.A. Harkness (ed.) Cross-Cultural Survey Equivalence, Mannheim: ZUMA, pp. 41–65.

    Google Scholar 

  • Vanhanen, T. (1997) Prospects of Democracy: A Study of 172 Countries, London: Routledge.

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank Theresia Smolka and the two reviewers for their helpful comments and suggestions.

Author information

Authors and Affiliations

  1. Department of Political Science, University of Duisburg-Essen, Forsthausweg 2, Duisburg, 47057, Germany

    susanne pickel, toralf stark & wiebke breustedt

Authors
  1. susanne pickel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. toralf stark
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. wiebke breustedt
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

pickel, s., stark, t. & breustedt, w. assessing the quality of quality measures of democracy: a theoretical framework and its empirical application. Eur Polit Sci 14, 496–520 (2015). https://doi.org/10.1057/eps.2015.61

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1057/eps.2015.61

Keywords

Search

Navigation