Skip to main content

Measuring Agreement in Ordered Rating Scales

Quality and Quantity volume 35, pages 325–341 (2001)Cite this article

Abstract

Ordered rating scales are one of the most frequently used question formats in large-scale surveys. Analysts of the responses to such questions often find themselves in need of describing the degree of agreement (concentration, consensus) of the answers to such questions. For that purpose they commonly use standard deviations of the response distributions, or measures based on these (such as the coefficient of consensus defined by Granberg and Holmberg, 1988), or the coefficient of variability, etc. This paper demonstrates that such measures are inappropriate for this purpose because they misrepresent what they are supposed to measure: the `peakedness' of a distribution. As an alternative a measure of agreement A is proposed. This measure is a weighted average of the degree of agreement that exists in the simple component parts – layers – into which any frequency distribution can be disaggregated, and for which agreement can be expressed in a straightforward and unequivocal way.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

References

  • Brug, W. van der (1997). Where's the Party? Voters' Perceptions of Party Positions. Doctoral Dissertation: University of Amsterdam.

  • Brug, W. van der & van der Eijk, C. (1999). The cognitive basis of voting. In: H. Schmitt & J. J. A. Thomassen (eds), Political Representation and Legitimacy in the European Union. Oxford: Oxford University Press, pp. 129-160.

    Google Scholar 

  • Converse, P. E. (1964). The nature of belief systems in mass publics. In: D. E. Apter (ed.), Ideology and Discontent. New York: Free Press, pp. 206-261.

    Google Scholar 

  • Converse, P. E. (1975). Public opinion and voting behavior. In: F. I. Greenstein & N.W. Polsby (eds), Handbook of Political Science, Vol. 4, Reading, MA: Addison Wesley, pp. 75-169.

    Google Scholar 

  • Converse, P. E. & Pierce, R. (1986). Political Representation in France. Cambridge, MA: Belknap Press.

    Google Scholar 

  • Coombs, C. H. (1964). A Theory of Data. New York: Wiley.

    Google Scholar 

  • Downs, A. (1957). An Economic Theory of Democracy. New York: Harper and Row.

    Google Scholar 

  • Eijk, C. van der (1990). Ideology, party systems and voting in Europe. Paper presented at Annual Meeting of the American Political Science Association, San Francisco, August 1990.

  • Eijk, C. van der & Franklin, M. N. (1996). Choosing Europe? The European Electorate and National Politics in the Face of Union. Ann Arbor: University of Michigan Press.

    Google Scholar 

  • Eijk, C. van der, Franklin, M. N. & van der Brug, W. (1999). Policy preferences and party choice. In: H. Schmitt & J. J. A. Thomassen (eds), Political Representation and Legitimacy in the European Union. Oxford: Oxford University Press, pp. 161-185.

    Google Scholar 

  • Eijk, C. van der, Niemöller, B. & Eggen, A. Th. J. (1981). Dutch Parliamentary Election Study 1981. Amsterdam: University of Amsterdam.

    Google Scholar 

  • Eijk, C. van der, Oppenhuis, E. V. & Schmitt, H. (1993). EES-89: European Election Study 1989. Data Description and Preliminary Documentation. Amsterdam: Steinmetz Archive/Swidoc.

    Google Scholar 

  • Granberg, D. & Holmberg, S. (1988). The Political System Matters. Social Psychology and Voting Behavior in Sweden and the United States. Cambridge: Cambridge University Press.

    Google Scholar 

  • Herrera, C., Herrera, R. & Smith, E. (1992). Public opinion and congressional representation. Public Opinion Quarterly 56: 185-205.

    Google Scholar 

  • Huber, J. & Bingham Powell, G. (1994). Congruence between citizens and policymakers in two visions of liberal democracy. World Politics 46: 291-326.

    Google Scholar 

  • McClosky, H. (1964). Consensus and ideology in American politics. American Political Science Review 58: 361-382.

    Google Scholar 

  • McClosky, H., Hoffman, P. J. & O'Hara, R. (1960). Issue conflict and consensus among party leaders and followers. American Political Science Review 54: 406-427.

    Google Scholar 

  • Schmitt, H., van der Eijk, C., Scholz, E. & Klein, M. (1996). European Election Study 1994. Data Description and Documentation. Mannheim/Cologne, Zentralarchiv für Empirische Sozialforschung/ European Election Study Research Group.

    Google Scholar 

  • Schuur, W. H. van (1984). Structure in Political Beliefs. A New Model for Stochastic Unfolding with Applications to European Party Activists. Amsterdam: CT Press.

    Google Scholar 

  • Schuur, W. H. van (1993). Nonparametric unfolding models for multicategory data. Political Analysis 4: 41-74.

    Google Scholar 

  • Thomassen, J. J. A. & Jennings, M. K. (1989). Party Systems, Party Differences and Homogeneity. Paper presented at IPSA/ISAWorkgroup on Elections and Parties, Paris, April 1989.

  • Thomassen, J. J. A. & Schmitt, H. (1999). Partisan structures in the European parliament. In: R. S. Katz & B. Wessels (eds), The European Parliament, the National Parliaments and European Integration. Oxford: Oxford University Press, pp. 129-148.

    Google Scholar 

  • Oppenhuis, E. V. (1995). Voting Behavior in Europe. A Comparative Analysis of Electoral Participation and Party Choice. Amsterdam: Het Spinhuis.

    Google Scholar 

  • Pomper, G.M. (1972). From confusion to clarity: Issues and American voters, 1956–1968. American Political Science Review 66: 415-428.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Political Science, and Amsterdam School of Communication Research (ASCoR), University of Amsterdam, The Netherlands

    Cees Van Der Eijk

Authors
  1. Cees Van Der Eijk
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Van Der Eijk, C. Measuring Agreement in Ordered Rating Scales. Quality & Quantity 35, 325–341 (2001). https://doi.org/10.1023/A:1010374114305

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1010374114305

  • descriptive statistics
  • measures of concentration
  • measures of dispersion
  • survey research