Quality & Quantity

, Volume 49, Issue 6, pp 2291–2306 | Cite as

An iterative expert survey approach for estimating parties’ policy positions

  • Kostas Gemenis


This article introduces the iterative expert survey approach in estimating parties’ policy positions. Methodologically, the proposed approach is based on the tradition of ‘judgemental’ coding in the content analysis of political text, and incorporates the idea of anonymous iteration among a panel of expert coders taken from the method known as ‘Delphi’. Anonymous iteration presents an effective way of reducing the random error, and potential bias arising from inter-expert/coder disagreement evident in other popular methods. I provide an empirical demonstration of the approach by estimating parties’ policy positions in the context of a voting advice application in Germany, and argue that the method has considerable potential to generate valid and reliable data on party positions cross-nationally and retrospectively.


Content analysis Expert judgement Delphi method  Political parties 



Earlier versions of this article were presented in a seminar at the Institute for Advanced Studies, Vienna (April 2014) and a VAA workshop at the University of Twente (November 2013). I would like to thank the participants of these events, and particularly Fernando Mendez, Emmanuel Sigalas, Markus Wagner, and Jonathan Wheatley, for their helpful comments. The usual disclaimer applies.


  1. Albright, J.J., Mair, P.: Does the number of parties to place affect the placement of parties? Results from an expert survey experiment. Elect. Stud. 30, 858–864 (2011)Google Scholar
  2. Armstrong, J.S.: How to make better forecasts and decisions: avoid face-to-face meetings. Foresight 5, 3–8 (2006)Google Scholar
  3. Benoit, K., Laver, M., Mikhaylov, S.: Treating words as data with error: uncertainty in text statements of policy positions. Am. J. Polit. Sci. 53, 495–513 (2009)CrossRefGoogle Scholar
  4. Bolger, F., Wright, G.: Reliability and validity in expert judgment. In: Wright, G., Bolger, F. (eds.) Expertise and Decision Support, pp. 47–76. Plenum, London (1992)CrossRefGoogle Scholar
  5. Booker, J.M., Meyer, M.A.: Sources and effects of interexpert correlation: an empirical study. IEEE Trans. Syst. Man Cybern. 18, 135–142 (1988)CrossRefGoogle Scholar
  6. Brambor, T., Clark, W.R., Golder, M.: Understanding interaction models: improving empirical analyses. Polit. Anal. 14, 63–82 (2006)CrossRefGoogle Scholar
  7. Braun, D., Mikhaylov, S., Schmitt, H.: European Parliament Election Study 2009, Manifesto Study [electronic file]. GESIS-Leibniz Institute for the Social Sciences [distributor] ZA5057 (2010). doi: 10.4232/1.10204
  8. Budge, I.: Expert judgements of party policy positions: uses and limitations in political research. Eur. J. Polit. Res. 37, 103–113 (2000)Google Scholar
  9. Conti, N., Memoli, V.: The multi-faceted nature of party-based Euroscepticism. Acta Polit. 47, 91–112 (2012)CrossRefGoogle Scholar
  10. Curini, L.: Experts’ political preferences and their impact on ideological bias: an unfolding analysis based on a Benoit-Laver expert survey. Party Polit. 16, 299–321 (2010)CrossRefGoogle Scholar
  11. Dalkey, N.C.: Toward a theory of group estimation. In: Linstone, H.A., Turoff, M. (eds.) The Delphi Method: Techniques and Applications, pp. 236–261. Addison Wesley, Reading (1975)Google Scholar
  12. Dalkey, N.C., Helmer, O.: An experimental application of the Delphi method to the use of experts. Manag. Sci. 9, 458–467 (1963)CrossRefGoogle Scholar
  13. Dalton, R.J., Beck, P.A., Huckfeldt, R.: Partisan cues and the media: information flows in the 1992 presidential election. Am. Polit. Sci. Rev. 92, 111–126 (1998)CrossRefGoogle Scholar
  14. de Lange, S.L.: A new winning formula? The programmatic appeal of the radical right. Party Polit. 13, 411–435 (2007)CrossRefGoogle Scholar
  15. De Swaan, A.: An empirical model of coalition formation as an n-person game of policy distance minimization. In: Groenings, S., Kelley, E.W., Leiserson, M. (eds.) The Study of Coalition Behavior, pp. 424–444. Holt, Rinehart and Winston. New York (1970)Google Scholar
  16. Dinas, E., Gemenis, K.: Measuring parties’ ideological positions with manifesto data: a critical evaluation of the competing methods. Party Polit. 16, 427–450 (2010)CrossRefGoogle Scholar
  17. Djouvas, C., Gemenis, K., Mendez, F., Wheatley, J.: SmartCoding: an online platform for estimating parties’ policy positions. Unpublished manuscript, available at (2014)
  18. Dolezal, M.: The design of the study: the distinguishing characteristics of our approach. In: Kriesi, H., Grande, E., Lachat, R., Dolezal, M., Bornschier, S., Frey, T. (eds.) West European Politics in the Age of Globalization, pp. 53–74. Cambridge University Press, Cambridge (2008)Google Scholar
  19. Dolezal, M., Ennser-Jedenastik, L., Müller, W.C., Winkler, A.K.: How parties compete for votes: a test of saliency theory. Eur. J. Polit. Res. 53, 57–76 (2014)CrossRefGoogle Scholar
  20. Downs, A.: An Economic Theory of Democracy. Harper and Brothers, New York (1957)Google Scholar
  21. Einhorn, H.J.: Expert judgment: some necessary conditions and an example. J. Appl. Psychol. 59, 562–571 (1974)CrossRefGoogle Scholar
  22. Ferrell, W.R.: Combining individual judgments. In: Wright, G. (ed.) Behavioral Decision Making, pp. 111–145. Plenum Press, London (1985)CrossRefGoogle Scholar
  23. Garzia, D., Marschall, S. (eds.): Matching Voters with Parties and Candidates. ECPR Press, Colchester (2014)Google Scholar
  24. Gemenis, K.: Proxy documents as a source of measurement error in the Comparative Manifestos Project. Elect. Stud. 31, 594–604 (2012a)Google Scholar
  25. Gemenis, K.: A new approach for estimating parties’ positions in voting advice applications. Paper presented at the XXVI congress of the Italian Political Science Association, Rome, 13–15 September 2012b.Google Scholar
  26. Gemenis, K.: What to do (and not to do) with the Comparative Manifestos Project data. Polit. Stud. 61(S1), 23–43 (2013a)Google Scholar
  27. Gemenis, K.: Estimating parties’ policy positions through voting advice applications: some methodological considerations. Acta Polit. 48, 268–295 (2013b)CrossRefGoogle Scholar
  28. Gemenis, K., Dinas, E.: Confrontation still? Examining parties’ policy positions in Greece. Comp. Eur. Polit. 8, 179–201 (2010)CrossRefGoogle Scholar
  29. Gemenis, K., Van Ham, C.: Comparing methods for estimating parties’ positions in voting advice applications. In: Garzia, D., Marschall, S. (eds.) Matching Voters with Parties and Candidates, pp. 33–47. ECPR Press, Colchester (2014)Google Scholar
  30. Golder, M., Stramski, J.: Ideological congruence and electoral institutions. Am. J. Polit. Sci. 54, 90–106 (2010)CrossRefGoogle Scholar
  31. Grimmer, J., Stewart, B.M.: Text as data: the promise and pitfalls of automatic content analysis methods for political texts. Polit. Anal. 21, 267–297 (2013)CrossRefGoogle Scholar
  32. Gudbrandsen, F.: Partisan influence on immigration: the case of Norway. Scand. Polit. Stud. 33, 248–270 (2010)CrossRefGoogle Scholar
  33. Hawkins, K.A.: Is Chávez populist? Measuring populist discourse in comparative perspective. Comp. Polit. Stud. 42, 1040–1067 (2009)CrossRefGoogle Scholar
  34. Janda, K.: Political Parties: A Cross-National Survey. Free Press, New York (1980)Google Scholar
  35. Jones Jr, R.J.: The state of presidential election forecasting: the 2004 experience. Int. J. Forecast. 24, 310–321 (2008)CrossRefGoogle Scholar
  36. Katakis, I., Tsapatsoulis, N., Mendez, F., Triga, V., Djouvas, C.: Social voting advice applications: definitions, challenges, datasets and evaluation. IEEE Trans. Syst. Man Cybern. 44, 1039–1052 (2014)Google Scholar
  37. Krippendorff, K.: Content Analysis: An Introduction to its Methodology, 2nd edn. Sage, Thousand Oaks (2004)Google Scholar
  38. Krouwel, A., van Elfrinkhof, A.: Combining strengths of methods of party positioning to counter their weaknesses: the development of a new methodology to calibrate parties on issues and ideological dimensions. Qual. Quant. 48, 1455–1472 (2014)CrossRefGoogle Scholar
  39. Krouwel, A., Vitiello, T., Wall, M.: The practicalities of issuing vote advice: a new methodology for profiling and matching. Int. J. Electron. Gov. 5, 223–243 (2012)CrossRefGoogle Scholar
  40. Krouwel, A., Vitiello, T., Wall, M.: Helping voters to vote? Dynamics created by VAAs in election campaigns between parties, voters and media. In: Garzia, D., Marschall, S. (eds.) Matching Voters with Parties and Candidates, pp. 67–78. ECPR Press, Colchester (2014)Google Scholar
  41. Lacewell, O.P., Werner, A.: Coder training: key to enhancing reliability and validity. In: Volkens, A., Bara, J., Budge, I., McDonald, I., Klingemann, H.-D. (eds.) Mapping Policy Preferences from Texts III, pp. 169–194. Oxford University Press, Oxford (2013)Google Scholar
  42. Lau, L.J., Frey, B.: Ideology, public approval, and government behavior. Public Choice 10, 21–40 (1971)CrossRefGoogle Scholar
  43. Laver, M.: Position and salience in the policies of political actors. In: Laver, M. (ed.) Estimating the Policy Position of Political Actors, pp. 66–75. Routledge, London (2001)Google Scholar
  44. Lewis, J.B., King, G.: No evidence on directional vs. proximity voting. Polit. Anal. 8, 21–33 (1999)CrossRefGoogle Scholar
  45. Lowe, W.: Understanding wordscores. Polit. Anal. 16, 356–371 (2008)CrossRefGoogle Scholar
  46. Mair, P.: Searching for the positions of political actors: a review of approaches and a critical evaluation of expert surveys. In: Laver, M. (ed.) Estimating the Policy Position of Political Actors, pp. 10–30. Routledge, London (2001)Google Scholar
  47. Meyer, M.A., Booker, J.M.: Eliciting and Analyzing Expert Judgment: A Practical Guide. Academic Press, London (1991)Google Scholar
  48. Mikhaylov, S., Laver, M., Benoit, K.: Coder reliability and misclassification in the human coding of party manifestos. Polit. Anal. 20, 78–91 (2012)CrossRefGoogle Scholar
  49. Mulgrave, N.W., Ducanis, A.J.: Propensity to change responses in a Delphi round as a function of dogmatism. In: Linstone, H.A., Turoff, M. (eds.) The Delphi Method: Techniques and Applications, pp. 288–290. Addison Wesley, Reading (1975)Google Scholar
  50. Müller, W.C., Strøm, K.: Coalition governance in Western Europe: an introduction. In: Müller, W.C., Strøm, K. (eds.) Coalition Governments in Western Europe, pp., pp. 1–31. Oxford University Press, Oxford (2003)Google Scholar
  51. Mumpower, J.L., Stewart, T.R.: Expert judgement and expert disagreement. Think. Reason. 2, 191–212 (1996)CrossRefGoogle Scholar
  52. Neuendorf, K.A.: The Content Analysis Guidebook. Sage, Thousand Oaks (2002)Google Scholar
  53. Odmalm, P.: Party competition and positions on immigration: strategic advantages and spatial locations. Comp. Eur. Polit. 10, 1–22 (2012)CrossRefGoogle Scholar
  54. Parenté, F.J., Anderson-Parenté, J.K.: Delphi inquiry systems. In: Wright, G., Ayton, P. (eds.) Judgmental Forecasting, pp. 129–156. Wiley, New York (1987)Google Scholar
  55. Pellikaan, H., Van der Meer, T., de Lange, S.L.: The road from a depoliticized to a centrifugal democracy. Acta Polit. 38, 23–49 (2003)CrossRefGoogle Scholar
  56. Riffe, D., Lacy, S., Fico, F.G.: Analyzing Media Messages: Using Quantitative Content Analysis in Research. Lawrence Erlbaum Associates, Mahwah (2005)Google Scholar
  57. Rowe, G.: Perspectives on expertise in the aggregation of judgments. In: Wright, G., Bolger, F. (eds.) Expertise and Decision Support, pp. 155–180. Plenum, New York (1992)CrossRefGoogle Scholar
  58. Rowe, G., Wright, G.: The Delphi technique as a forecasting tool: issues and analysis. Int. J. Forecast. 15, 353–375 (1999)CrossRefGoogle Scholar
  59. Ruedin, D.: Obtaining party positions on immigration in Switzerland: comparing different methods. Swiss Polit. Sci. Rev. 19, 84–105 (2013)CrossRefGoogle Scholar
  60. Scheibe, M., Skutsch, M., Schofer, J.: Experiments in Delphi methodology. In: Linstone, H.A., Turoff, M. (eds.) The Delphi Method: Techniques and Applications, pp. 262–287. Addison Wesley, Reading (1975)Google Scholar
  61. Shanteau, J., Hall, B.: What does it mean when experts disagree. In: Salas, E., Klein, G. (eds.) Linking Expertise and Naturalistic Decision Making, pp. 229–244. Lawrence Erlbaum Associates, Mahwah (2001)Google Scholar
  62. Shikano, S.: Estimating ideological positions of political parties using a deliberative expert survey. Paper presented at the voting advice applications workshop, University of Twente, The Netherlands, 15 November 2013.Google Scholar
  63. Shikano, S., Busemeyer, M., Djouvas, C., Elff, M., Gemenis, K., Mendez, F., Wheatley, J.: ParteieNavi: voting advice application data for the 2013 German federal election [electronic file]. GESIS-Leibniz Institute for the Social Sciences [distributor] (2014). doi: 10.7802/68
  64. Steenbergen, M.R., Marks, G.: Evaluating expert judgments. Eur. J. Polit. Res. 46, 347–366 (2007)CrossRefGoogle Scholar
  65. Stokes, D.E.: Spatial models of party competition. Am. Polit. Sci. Rev. 57, 368–377 (1963)CrossRefGoogle Scholar
  66. Tilley, J., Wlezien, C.: Does political information matter? An experimental test relating to party positions on Europe. Polit. Stud. 56, 56–214 (2008)CrossRefGoogle Scholar
  67. Trechsel, A.H., Mair, P.: When parties (also) position themselves: an introduction to the EU Profiler. J. Inf. Technol. Polit. 8, 1–20 (2011)CrossRefGoogle Scholar
  68. Tversky, A., Kahneman, D.: Judgment under uncertainty: heuristics and biases. Science 185, 1124–1131 (1974)CrossRefGoogle Scholar
  69. Volkens, A.: Strengths and weaknesses of approaches to measuring policy positions of parties. Elect. Stud. 26, 108–120 (2007)CrossRefGoogle Scholar
  70. Walter, A.S., van der Brug, W., van Praag, P.: When the stakes are high: party competition and negative campaigning. Comp. Polit. Stud. 47, 550–573 (2014)Google Scholar
  71. Zulianello, M.: Analyzing party competition through the comparative manifesto data: some theoretical and methodological considerations. Qual. Quant. 48, 1723–1737 (2014)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2014

Authors and Affiliations

  1. 1.Department of Public AdministrationUniversity of TwenteEnschedeThe Netherlands

Personalised recommendations