Quality & Quantity

, Volume 50, Issue 3, pp 981–1007 | Cite as

Dynamic scale validation reloaded

Assessing the psychometric properties of latent measures of ideology in VAA spatial maps
  • Micha Germann
  • Fernando Mendez


Voting advice applications (VAAs), online tools that provide voters with an estimate of their ideological congruence with political parties or candidates, have become increasingly popular in recent years. Many VAAs draw on low-dimensional spatial representations to match voters to political elites. Yet VAA spatial maps tend to be defined purely on a priori grounds. Thus fundamental psychometric properties, such as unidimensionality and reliability, remain unchecked and potentially violated. This practice can be damaging to the quality of spatial matches. In this paper we propose dynamic scale validation (DSV) as a method to empirically validate and thereby improve VAA spatial maps. The basic logic is to draw on data generated by users who access the VAA soon after its launch for an evaluation (and potential adjustment) of the spatial maps. We demonstrate the usefulness of DSV drawing on data from three actual VAAs: ParteieNavi, votulmeu and choose4cyprus.


Voting advice applications Spatial maps Psychometrics Mokken scale analysis Unidimensionality Reliability 



We thank L. Andries van der Ark, Hendrik Straat, Kostas Gemenis, Jonas Lefevere, Simon Otjes, Jonathan Wheatley, the anonymous reviewer and seminar participants at the 2013 ECPR General Conference (in Bordeaux) and the 2013 EU Vox workshop (in Twente) for helpful comments. Any remaining errors are, of course, our own. The authors gratefully acknowledge financial support from the e-Democracy project funded by the Swiss cantons of Argovia (main contributor), Basel-City, Geneva, Grisons and Schaffhausen as well as the Swiss Federal Chancellery. We used R v.3.1 for much of the scaling analysis, in particular the mokken (van der Ark 2007, 2012) and the poLCA (Linzer and Lewis 2011) packages, and Stata v.11.2 for all remaining analyses. Replication files are available on User’s data privacy is fully protected.

Supplementary material

11135_2015_186_MOESM1_ESM.pdf (14 kb)
Supplementary material (pdf 15 kb)


  1. Achen, C.H.: Mass political attitudes survey response. Am. Polit. Sci. Rev. 69(4), 1218–1231 (1975)CrossRefGoogle Scholar
  2. Andreadis, I.: Data quality and data cleaning. In: Garzia, D., Marschall, S. (eds.) Matching Voters with Parties and Candidates, pp. 79–92. ECPR Press, Colchester (2014)Google Scholar
  3. Ansolabehere, S., Rodden, J., Snyder, J.M.: The strength of issues: using multiple measures to gauge preference stability, ideological constraint, and issue voting. Am. Polit. Sci. Rev. 102(2), 215–232 (2008)CrossRefGoogle Scholar
  4. Benoit, K., Laver, M.: The dimensionality of political space: epistemological and methodological considerations. Eur. Union Polit. 13(2), 194–218 (2012)CrossRefGoogle Scholar
  5. Carmines, E.G., Zeller, R.A.: Reliability and Validity Assessment. Sage Publications, Thousand Oaks, CA (1979)CrossRefGoogle Scholar
  6. Cedroni, L., Garzia, D. (eds.): Voting Advice Applications in Europe: The State of the Art. ScriptaWeb, Naples (2010)Google Scholar
  7. Clark, L.A., Watson, D.: Constructing validity: basic issues in objective scale development. Psychol. Assess. 7(3), 309–319 (1995)CrossRefGoogle Scholar
  8. Converse, P.E.: The nature of belief systems in mass publics. In: Apter, D.E. (ed.) Ideology and Discontent, pp. 206–261. Free Press, New York, NY (1964)Google Scholar
  9. Cortina, J.M.: What is coefficient alpha? An examination of theory and applications. J. Appl. Psychol. 78(1), 98–104 (1993)CrossRefGoogle Scholar
  10. Davidov, E.: Measurement equivalence of nationalism and constructive patriotism in the ISSP: 34 Countries in a comparative perspective. Polit. Anal. 17(1), 64–82 (2009)CrossRefGoogle Scholar
  11. Gadermann, A.M., Guhn, M., Zumbo, B.D.: Estimating ordinal reliability for Likert-type and ordinal item response data: a conceptual, empirical, and practical guide. Pract. Assess. Res. Eval. 17(3), 1–13 (2012)Google Scholar
  12. Garzia, D., Angelis, Ad, Pianzola, J.: The impact of voting advice applications on electoral participation. In: Garzia, D., Marschall, S. (eds.) Matching Voters with Parties and Candidates, pp. 105–114. ECPR Press, Colchester (2014)Google Scholar
  13. Gemenis, K.: Estimating parties’ policy positions through voting advice applications: some methodological considerations. Acta Polit. 48(3), 268–295 (2013)CrossRefGoogle Scholar
  14. Gemenis, K., Rosema, M.: Voting advice applications and electoral turnout. Elect. Stud. 36, 281–289 (2014). doi: 10.1016/j.electstud.2014.06.010
  15. Gemenis, K., van Ham, C.: Comparing methods for estimating parties’ positions in voting advice applications. In: Garzia, D., Marschall, S. (eds.) Matching Voters with Parties and Candidates, pp. 33–47. ECPR Press, Colchester (2014)Google Scholar
  16. Gerbing, D.W., Anderson, J.C.: An updated paradigm for scale development incorporating unidimensionality and its assessment. J. Mark. Res. 25(2), 186–192 (1988)CrossRefGoogle Scholar
  17. Germann, M., Mendez, F., Wheatley, J., Serdült, U.: Exploiting smartvote data for the ideological mapping of Swiss political parties. Paper presented at the 2012 Convegno SISP, September 13–15, Rome (2012)Google Scholar
  18. Germann, M., Mendez, F., Wheatley, J., Serdült, U.: Spatial maps in voting advice applications: the case for dynamic scale validation. Acta Politica (2014). doi: 10.1057/ap.2014.3
  19. Hattie, J.: Methodology review: assessing unidimensionality of tests and items. Appl. Psychol. measurement 9(2), 139–164 (1985)CrossRefGoogle Scholar
  20. Hemker, B.T., Sijtsma, K., Molenaar, I.W.: Selection of unidimensional scales from a multidimensional item bank in the polytomous Mokken IRT model. Appl. Psychol. Measurement 19(4), 337–352 (1995)CrossRefGoogle Scholar
  21. Kriesi, H., Grande, E., Lachat, R., Dolezal, M., Bornschier, S., Frey, T.: Globalization and the transformation of the national political space: six European countries compared. Eur. J. Polit. Res. 45(6), 921–956 (2006)CrossRefGoogle Scholar
  22. Lefevere, J., Walgrave, S.: A perfect match? The impact of statement selection on voting advice applications’ ability to match voters and parties. Elect. Stud. 36, 252–262 (2014). doi: 10.1016/j.electstud.2014.04.002
  23. Lin, L.I.K.: A concordance correlation coefficient to evaluate reproducibility. Biometrics 45(1), 255 (1989)CrossRefGoogle Scholar
  24. Linzer, D.A., Lewis, J.B.: poLCA: an R package for polytomous variable latent class analysis. J. Stat. Softw. 42(10), 1–29 (2011)CrossRefGoogle Scholar
  25. Lobo, M.C., Vink, M., Lisi, M.: Mapping the political landscape: a vote advice application in Portugal. In: Cedroni, L., Garzia, D. (eds.) Voting Advice Applications in Europe, pp. 143–172. ScriptaWeb, Naples (2010)Google Scholar
  26. Lord, F.M., Novick, M.R.: Statistical Theories of Mental Test Scores. Addison-Wesley, Reading (1968)Google Scholar
  27. Louwerse, T., Otjes, S.: Design challenges in cross-national VAAs: the case of the EU profiler. Int. J. Electron. Gov. 5(3/4), 279–297 (2012)CrossRefGoogle Scholar
  28. Louwerse, T., Rosema, M.: The design effects of voting advice applications: comparing methods of calculating matches. Acta Polit. 49(3), 286–312 (2014)CrossRefGoogle Scholar
  29. Marks, G., Hooghe, L., Nelson, M., Edwards, E.: Party competition and European integration in the east and west: different structure, same causality. Comp. Polit. Stud. 39(2), 155–175 (2006)CrossRefGoogle Scholar
  30. Marschall, S.: Profiling users. In: Garzia, D., Marschall, S. (eds.) Matching Voters with Parties and Candidates, pp. 93–104. ECPR Press, Colchester (2014)Google Scholar
  31. Marschall, S., Schultze, M.: Voting advice applications and their effect on voter turnout: the case of the German Wahl-O-Mat. Int. J. Electron. Gov. 5(3/4), 349–366 (2012)CrossRefGoogle Scholar
  32. McDonald, R.P.: Generalizability in factorable domains: domain validity and generalizability. Educ. Psychol. Meas. 38(1), 75–79 (1978)CrossRefGoogle Scholar
  33. Mendez, F.: Matching voters with political parties and candidates: an empirical test of four algorithms. Int. J. Electron. Gov. 5(3/4), 264–278 (2012)CrossRefGoogle Scholar
  34. Mendez, F.: What’s behind a matching algorithm? A critical assessment of how voting advice applications produce voting recommendations. In: Garzia, D., Marschall, S. (eds.) Matching Voters with Parties and Candidates, pp. 49–66. ECPR Press, Colchester (2014)Google Scholar
  35. Mokken, R.J.: A Theory and Procedure of Scale Analysis Applications in Political Research. Springer, New York (1971)CrossRefGoogle Scholar
  36. O’Leary-Kelly, S., Vokurka, R.J.: The empirical assessment of construct validity. J. Oper. Manag. 16(4), 387–405 (1998)CrossRefGoogle Scholar
  37. Otjes, S., Louwerse, T.: Spatial models in voting advice applications. Elect. Stud. 36, 263–271 (2014). doi: 10.1016/j.electstud.2014.04.004
  38. Sijtsma, K.: On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika 74(1), 107–120 (2009)CrossRefGoogle Scholar
  39. Smits, I.A.M., Timmerman, M.E., Meijer, R.R.: Exploratory Mokken scale analysis as a dimensionality assessment tool: why scalability does not imply unidimensionality. Appl. Psychol. Meas. 36(6), 516–539 (2012)CrossRefGoogle Scholar
  40. Straat, J.H., van der Ark, L.A., Sijtsma, K.: Comparing optimization algorithms for item selection in Mokken scale analysis. J. Classif. 30(1), 75–99 (2013)CrossRefGoogle Scholar
  41. Straat, J.H., van der Ark, L.A., Sijtsma, K.: Minimum sample size requirements for Mokken scale analysis. Educ. Psychol. Meas. 74(5), 809–822 (2014)CrossRefGoogle Scholar
  42. van Camp, K., Lefevere, J., Walgrave, S.: The content and formulation of statements in voting advice applications: a comparative analysis of 26 VAAs. In: Garzia, D., Marschall, S. (eds.) Matching Voters with Parties and Candidates, pp. 11–31. ECPR Press, Colchester (2014)Google Scholar
  43. van de Pol, J., Holleman, B., Kamoen, N., Krouwel, A., de Vreese, C.: Beyond young, highly educated males: a typology of VAA users. J. Inf. Technol. Polit. 11(4), 397–411 (2014)CrossRefGoogle Scholar
  44. van der Ark, L.A.: Mokken scale analysis in R. J. Stat. Softw. 20(11), 1–19 (2007)CrossRefGoogle Scholar
  45. van der Ark, L.A.: New developments in Mokken scale analysis in R. J. Stat. Softw. 48(5), 1–27 (2012)CrossRefGoogle Scholar
  46. van der Ark, L.A., van der Palm, D.W., Sijtsma, K.: A latent class approach to estimating test-score reliability. Appl. Psychol. Meas. 35(5), 380–392 (2011)CrossRefGoogle Scholar
  47. van Schuur, W.H.: Mokken scale analysis: between the Guttman scale and parametric item response theory. Polit. Anal. 11(2), 139–163 (2003)CrossRefGoogle Scholar
  48. Vassil, K.: Voting Smarter: The Impact of Voting Advice Applications on Political Behavior. PhD thesis, European University Institute, Florence (2012)Google Scholar
  49. Wagner, M., Ruusuvirta, O.: Matching voters to parties: voting advice applications and models of party choice. Acta Polit. 47(4), 400–422 (2012)CrossRefGoogle Scholar
  50. Walgrave, S., Nuytemans, M., Pepermans, K.: Voting aid applications and the effect of statement selection. West Eur. Polit. 32(6), 1161–1180 (2009)CrossRefGoogle Scholar
  51. Wall, M., Krouwel, A., Vitiello, T.: Dutch legislative elections. Party Polit. 20(3), 416–428 (2010)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2015

Authors and Affiliations

  1. 1.Centre for Democracy Studies Aarau (ZDA)University of ZurichAarauSwitzerland
  2. 2.Centre for Comparative and International StudiesETH ZurichZurichSwitzerland

Personalised recommendations