Quality and Quantity

, Volume 35, Issue 3, pp 325–341 | Cite as

Measuring Agreement in Ordered Rating Scales

  • Cees Van Der Eijk


Ordered rating scales are one of the most frequently used question formats in large-scale surveys. Analysts of the responses to such questions often find themselves in need of describing the degree of agreement (concentration, consensus) of the answers to such questions. For that purpose they commonly use standard deviations of the response distributions, or measures based on these (such as the coefficient of consensus defined by Granberg and Holmberg, 1988), or the coefficient of variability, etc. This paper demonstrates that such measures are inappropriate for this purpose because they misrepresent what they are supposed to measure: the `peakedness' of a distribution. As an alternative a measure of agreement A is proposed. This measure is a weighted average of the degree of agreement that exists in the simple component parts – layers – into which any frequency distribution can be disaggregated, and for which agreement can be expressed in a straightforward and unequivocal way.

descriptive statistics measures of concentration measures of dispersion survey research 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Brug, W. van der (1997). Where's the Party? Voters' Perceptions of Party Positions. Doctoral Dissertation: University of Amsterdam.Google Scholar
  2. Brug, W. van der & van der Eijk, C. (1999). The cognitive basis of voting. In: H. Schmitt & J. J. A. Thomassen (eds), Political Representation and Legitimacy in the European Union. Oxford: Oxford University Press, pp. 129-160.Google Scholar
  3. Converse, P. E. (1964). The nature of belief systems in mass publics. In: D. E. Apter (ed.), Ideology and Discontent. New York: Free Press, pp. 206-261.Google Scholar
  4. Converse, P. E. (1975). Public opinion and voting behavior. In: F. I. Greenstein & N.W. Polsby (eds), Handbook of Political Science, Vol. 4, Reading, MA: Addison Wesley, pp. 75-169.Google Scholar
  5. Converse, P. E. & Pierce, R. (1986). Political Representation in France. Cambridge, MA: Belknap Press.Google Scholar
  6. Coombs, C. H. (1964). A Theory of Data. New York: Wiley.Google Scholar
  7. Downs, A. (1957). An Economic Theory of Democracy. New York: Harper and Row.Google Scholar
  8. Eijk, C. van der (1990). Ideology, party systems and voting in Europe. Paper presented at Annual Meeting of the American Political Science Association, San Francisco, August 1990.Google Scholar
  9. Eijk, C. van der & Franklin, M. N. (1996). Choosing Europe? The European Electorate and National Politics in the Face of Union. Ann Arbor: University of Michigan Press.Google Scholar
  10. Eijk, C. van der, Franklin, M. N. & van der Brug, W. (1999). Policy preferences and party choice. In: H. Schmitt & J. J. A. Thomassen (eds), Political Representation and Legitimacy in the European Union. Oxford: Oxford University Press, pp. 161-185.Google Scholar
  11. Eijk, C. van der, Niemöller, B. & Eggen, A. Th. J. (1981). Dutch Parliamentary Election Study 1981. Amsterdam: University of Amsterdam.Google Scholar
  12. Eijk, C. van der, Oppenhuis, E. V. & Schmitt, H. (1993). EES-89: European Election Study 1989. Data Description and Preliminary Documentation. Amsterdam: Steinmetz Archive/Swidoc.Google Scholar
  13. Granberg, D. & Holmberg, S. (1988). The Political System Matters. Social Psychology and Voting Behavior in Sweden and the United States. Cambridge: Cambridge University Press.Google Scholar
  14. Herrera, C., Herrera, R. & Smith, E. (1992). Public opinion and congressional representation. Public Opinion Quarterly 56: 185-205.Google Scholar
  15. Huber, J. & Bingham Powell, G. (1994). Congruence between citizens and policymakers in two visions of liberal democracy. World Politics 46: 291-326.Google Scholar
  16. McClosky, H. (1964). Consensus and ideology in American politics. American Political Science Review 58: 361-382.Google Scholar
  17. McClosky, H., Hoffman, P. J. & O'Hara, R. (1960). Issue conflict and consensus among party leaders and followers. American Political Science Review 54: 406-427.Google Scholar
  18. Schmitt, H., van der Eijk, C., Scholz, E. & Klein, M. (1996). European Election Study 1994. Data Description and Documentation. Mannheim/Cologne, Zentralarchiv für Empirische Sozialforschung/ European Election Study Research Group.Google Scholar
  19. Schuur, W. H. van (1984). Structure in Political Beliefs. A New Model for Stochastic Unfolding with Applications to European Party Activists. Amsterdam: CT Press.Google Scholar
  20. Schuur, W. H. van (1993). Nonparametric unfolding models for multicategory data. Political Analysis 4: 41-74.Google Scholar
  21. Thomassen, J. J. A. & Jennings, M. K. (1989). Party Systems, Party Differences and Homogeneity. Paper presented at IPSA/ISAWorkgroup on Elections and Parties, Paris, April 1989.Google Scholar
  22. Thomassen, J. J. A. & Schmitt, H. (1999). Partisan structures in the European parliament. In: R. S. Katz & B. Wessels (eds), The European Parliament, the National Parliaments and European Integration. Oxford: Oxford University Press, pp. 129-148.Google Scholar
  23. Oppenhuis, E. V. (1995). Voting Behavior in Europe. A Comparative Analysis of Electoral Participation and Party Choice. Amsterdam: Het Spinhuis.Google Scholar
  24. Pomper, G.M. (1972). From confusion to clarity: Issues and American voters, 1956–1968. American Political Science Review 66: 415-428.Google Scholar

Copyright information

© Kluwer Academic Publishers 2001

Authors and Affiliations

  • Cees Van Der Eijk
    • 1
  1. 1.Department of Political Science, and Amsterdam School of Communication Research (ASCoR)University of AmsterdamThe Netherlands

Personalised recommendations