Multilevel Models for the Analysis of Comparative Survey Data: Common Problems and Some Solutions

  • Alexander W. Schmidt-CatranEmail author
  • Malcolm Fairbrother
  • Hans-Jürgen Andreß


This paper provides an overview over the application of mixed models (multilevel models) to comparative survey data where the context units of interest are countries. Such analyses have gained much popularity in the last two decades but they also come with a variety of challenges, some of which are discussed here. A focus lies on the small-N problem, influential cases (outliers) and the issue of omitted variables at the country level. Summarizing the methodological literature, the paper provides recommendations for applied researchers when possible or otherwise points to the more detailed literature. Some solutions for the small-N problem and omitted variable bias are discussed in detail, recommending the pooling of multiple survey waves to increase statistical power and to allow for the estimation of within-country effects, thereby controlling for unobserved heterogeneity. All issues are illustrated using an empirical example with data from the European Social Survey. The online appendix provides detailed syntax to adopt the presented procedures to researchers’ own data.


Mixed models Multilevel models Small-N problem Influential cases Omitted variable bias 

Mehrebenenmodelle zur Analyse von vergleichenden Umfragedaten: Häufige Probleme und ausgewählte Lösungsansätze


Die vorliegende Arbeit bietet einen Überblick über die Anwendung von Mehrebenenmodellen auf international vergleichende Umfragedaten. Mehrebenenanalysen, in denen die relevanten Kontexteinheiten Länder sind, haben in den letzten 2 Jahrzehnten eine weite Verbreitung gefunden, sind allerdings aus statistischer Perspektive in einigen Aspekten problematisch. Dieser Artikel zielt auf einige der Probleme ab, die bei der Anwendung von Mehrebenenanalysen auf internationale Umfragedaten auftreten. Ein Fokus liegt dabei auf dem small-N-Problem, einflussreichen Fällen („Ausreißern“) und dem Problem unbeobachteter Heterogenität auf der Länderebene. Dieser Beitrag bietet eine Zusammenfassung der methodischen Literatur zu Mehrebenenmodellen und versucht, in Forschung Tätigen möglichst konkrete Empfehlungen zu geben oder – wo dies nicht möglich ist – auf die tiefergehende Literatur zu verweisen. Lösungsansätze für das small-N-Problem und das Problem unbeobachteter Heterogenität werden im Detail diskutiert. Aus dieser Diskussion ergibt sich die Empfehlung, vorhandene Wellen international vergleichender Umfragedaten zu poolen. Zur Illustration verwendet dieser Artikel ein empirisches Beispiel auf Basis der Daten des European Social Survey. Der Online-Anhang enthält zu diesen Beispielen eine detaillierte Syntax, die sich leicht für andere Daten und Forschungsfragen anpassen lässt.


Gemischtes Modell Mehrebenenmodelle Small-N-Problem Ausreißer Unbeobachtete Heterogenität 


  1. Allison, Paul D. 2009. Fixed effects regression models. Thousand Oaks: SAGE.CrossRefGoogle Scholar
  2. Andress, Hans-Jürgen, Katrin Golsch and Alexander W. Schmidt. 2013. Applied panel data analysis for economic and social surveys. Springer: Berlin, Heidelberg.CrossRefGoogle Scholar
  3. Arceneaux, Kevin, and Gregory A. Huber. 2007. What to do (and not do) with multicollinearity in state politics research. State Politics & Policy Quarterly 7:81–101.CrossRefGoogle Scholar
  4. Babones, Salvatore J. 2013. Methods for quantitative macro-comparative research. London: Sage.Google Scholar
  5. Barr, Dale J., Roger Levy, Christoph Scheepers and Harry J. Tily. 2013. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language 68:255–278.CrossRefGoogle Scholar
  6. Beck, Nathaniel. 2001. Time-series–cross-section data: What have we learned in the past few years? Annual Review of Political Science 4:271–293.CrossRefGoogle Scholar
  7. Bell, Andrew J., and Kelvyn Jones. 2015. Explaining fixed effects: Random effects modeling of time-series cross-sectional and data panel data. Political Science Research and Methods 3:133–153.CrossRefGoogle Scholar
  8. Bell, Andrew J., Kelvyn Jones and Malcolm Fairbrother. 2018. Understanding and misunderstanding Group mean centering: A commentary on Kelley et al.’s dangerous practice. Quality & Quantity 52:2031–2046.CrossRefGoogle Scholar
  9. Bell, Andrew, Malcolm Fairbrother and Kelvyn Jones. 2019. Fixed and random effects models: Making an informed choice. Quality & Quantity 53:1051–1074.CrossRefGoogle Scholar
  10. Bell, Bethany A., Grant B. Morgan, Jason A. Schoeneberger, Jeffrey D. Kromrey and John M. Ferron. 2014. How low can you go? Methodology 10:1–11.CrossRefGoogle Scholar
  11. Belsley, David A., Edwin Kuh and Roy E. Welsch. 1980. Regression diagnostics: Identifying Influential Data and Sources of Collinearity. John Wiley: New York.CrossRefGoogle Scholar
  12. Bowers, Jake, and Katherine W. Drake. 2005. EDA for HLM: Visualization when probabilistic inference fails. Political Analysis 13:301–326.CrossRefGoogle Scholar
  13. Brännström, Lars. 2008. Making their mark: The effects of neighbourhood and upper secondary school on educational achievement. European Sociological Review 24:463–478.CrossRefGoogle Scholar
  14. Browne, William J., and David Draper. 2000. Implementation and performance issues in the Bayesian and likelihood fitting of multilevel models. Computational Statistics 15:391–420.CrossRefGoogle Scholar
  15. Bryan, Mark L., and Stephen P. Jenkins. 2016. Multilevel modelling of country effects: A cautionary tale. European Sociological Review 32:3–22.CrossRefGoogle Scholar
  16. Cook, R. Dennis. 1977. Detection of influential observation in linear regression. Technometrics 19:15–18.Google Scholar
  17. Draper, David. 2008. Bayesian multilevel analysis and MCMC. In Handbook of multilevel analysis, eds. Jan De Leeuw and Erik Meijer, 77–139. Springer: New York.CrossRefGoogle Scholar
  18. Ebbinghaus, Bernhard. 2005. When less is more: selection problems in large-N and small-N cross-national comparisons. International Sociology 20:133–152.CrossRefGoogle Scholar
  19. Elff, Martin, Jan P. Heisig, Merlin Schaeffer and Susumu Shikano. 2016. No need to turn Bayesian in multilevel analysis with few clusters: How frequentist methods provide unbiased estimates and accurate inference. Open Science Framework ( Scholar
  20. Elwert, Felix. 2013. Graphical causal models. In Handbook of causal analysis for social research, ed. Morgan, Stephen L., 245–273. Springer Netherlands: Dordrecht.CrossRefGoogle Scholar
  21. Enders, Craig K., and Davood Tofighi. 2007. Centering predictor variables in cross-sectional multilevel models: A new look at an old issue. Psychological Methods 12:121–138.CrossRefGoogle Scholar
  22. European Social Survey Cumulative File, ESS 1–7 (2016). Data file edition 1.0. NSD – Norwegian Centre for Research Data, Norway – Data Archive and distributor of ESS data for ESS ERIC.Google Scholar
  23. Fairbrother, Malcolm. 2013. Rich people, poor people, and environmental concern: Evidence across nations and time. European Sociological Review 29:910–922.CrossRefGoogle Scholar
  24. Fairbrother, Malcolm. 2014. Two multilevel modeling techniques for analyzing comparative longitudinal survey datasets. Political Science Research and Methods 2:119–140.CrossRefGoogle Scholar
  25. Fairbrother, Malcolm. 2016. Trust and public support for environmental protection in diverse national contexts. Sociological Science 3: 359–382.CrossRefGoogle Scholar
  26. Finseraas, H. 2012. Poverty, ethnic minorities among the poor, and preferences for redistribution in European regions. Journal of European Social Policy 22:164–180.CrossRefGoogle Scholar
  27. Fontaine, Johnny R.J. 2015. Traditional and multilevel approaches in cross-cultural research: An integration of methodological frameworks. In Multilevel analysis of individuals and cultures, eds. Van de Vijver, Fons J.R., Dianne A. Van Hemert and Ype H. Poortinga, 65–93. New York: Psychology Press.Google Scholar
  28. Gelman, Andrew. 2005. Two-stage regression and multilevel modeling: A commentary. Political Analysis 13:459–61.CrossRefGoogle Scholar
  29. Gelman, Andrew. 2006. Prior distributions for variance parameters in hierarchical models. Bayesian Analysis 1:72–91.CrossRefGoogle Scholar
  30. Gelman, Andrew, and Jennifer Hill. 2007. Data analysis using regression and multilevel/hierarchical models. New York: Cambridge University Press.Google Scholar
  31. Giesselmann, Marco and Alexander W. Schmidt-Catran. 2018. Getting the within estimator of cross-level interactions in multilevel models with pooled cross-sections: Why country dummies (sometimes) do not do the job. Sociological Methodology. Google Scholar
  32. Giger, Nathalie. 2012. Is social policy retrenchment unpopular? How welfare reforms affect government popularity. European Sociological Review 28:691–700.CrossRefGoogle Scholar
  33. Goerres, Achim, Markus B. Siewert, and Claudius Wagemann. 2019. Internationally comparative research designs in the social sciences: Fundamental issues, case selection logics, and research limitations. In Cross-national comparative research – analytical strategies, results and explanations. Sonderheft Kölner Zeitschrift für Soziologie und Sozialpsychologie. Eds. Hans-Jürgen Andreß, Detlef Fetchenhauer and Heiner Meulemann. Wiesbaden: Springer VS.
  34. Goldthorpe, John H. 1997. Current issues in comparative macrosociology: A debate on methodological issues. Comparative Social Research 16:1–26.Google Scholar
  35. Te Grotenhuis, Manfred, Marijn Scholte, Nan Dirk de Graaf and Ben Pelzer. 2015. The between and within effects of social security on church attendance in Europe 1980–1998: The danger of testing hypotheses cross-nationally. European Sociological Review 31:643–654.CrossRefGoogle Scholar
  36. Heisig, Jan P., Merlin Schaeffer and Johannes Giesecke. 2017. The costs of simplicity: Why multilevel models may benefit from accounting for cross-cluster differences in the effects of controls. American Sociological Review 82:796–827.CrossRefGoogle Scholar
  37. Heisig, Jan Paul, and Merlin Schaeffer. 2019. Why You Should Always Include a Random Slope for the Lower-Level Variable Involved in a Cross-Level Interaction. European Sociological Review 35:258–279.CrossRefGoogle Scholar
  38. Hox, Joop J. 2010. Multilevel analysis: Techniques and applications, 2nd Edition. New York: Routledge.CrossRefGoogle Scholar
  39. Immerzeel, Tim, and Frank Van Tubergen. 2013. Religion as reassurance? Testing the insecurity theory in 26 European countries. European Sociological Review 29:359–372.CrossRefGoogle Scholar
  40. Jackman, Simon D. 2009. Bayesian analysis for the social sciences. New York: John Wiley.CrossRefGoogle Scholar
  41. Jaeger, Mads M. 2013. The effect of macroeconomic and social conditions on the demand for redistribution: A pseudo panel approach. Journal of European Social Policy 23:149–163.CrossRefGoogle Scholar
  42. Kim, Jee-Seon, and Edward W. Frees. 2006. Omitted variables in multilevel models. Psychometrika 71:659–690.CrossRefGoogle Scholar
  43. Maas, Cora J. M., and Joop J. Hox. 2005. Sufficient sample sizes for multilevel modeling. methodology: European Journal of Research Methods for the Behavioral and Social Sciences 1:86–92.CrossRefGoogle Scholar
  44. Van der Meer, Tom, Manfred Te Grotenhuis and Ben Pelzer. 2010. Influential cases in multilevel modeling: A methodological comment. American Sociological Review 75:173–178.CrossRefGoogle Scholar
  45. Mewes, Jan. 2014. Gen (d) eralized trust: women, work, and trust in strangers. European Sociological Review 30:373–386.CrossRefGoogle Scholar
  46. Möhring, Katja, and Alexander W. Schmidt. 2013. MLT: Stata module to provide multilevel tools (Statistical Software Components S457577). Boston, MA: Boston College Department of Economics.Google Scholar
  47. Mundlak, Yair. 1978. Pooling of time-series and cross-section data. Econometrica 46:69–85.CrossRefGoogle Scholar
  48. Patterson, H. Desmond, and Robin Thompson. 1971. Recovery of inter-block information when block sizes are unequal. Biometrika 58:545–554.CrossRefGoogle Scholar
  49. Rabe-Hesketh, Sophia, and Anders Skrondal. 2012. Multilevel and longitudinal modeling using stata, 3rd Edition. College Station, TX: Stata Press.Google Scholar
  50. Schmidt-Catran, Alexander W. 2016. Economic inequality and demand for redistribution: Cross-sectional and longitudinal evidence. Socio-Economic Review 14:119–140.CrossRefGoogle Scholar
  51. Schmidt-Catran, Alexander W., and Malcom Fairbrother. 2016. The random effects in multilevel models: Getting them wrong and getting them right. European Sociological Review 32:23–38.CrossRefGoogle Scholar
  52. Semyonov, Moshe, Rebeca Raijman and Anastasia Gorodzeisky. 2006. The rise of anti-foreigner sentiment in European societies, 1988–2000. American Sociological Review 71:426–449.CrossRefGoogle Scholar
  53. Snijders, Tom A.B., and Johannes Berkhof. 2008. Diagnostic checks for multilevel models. In Handbook of multilevel analysis, eds. Jan De Leeuw and Erik Meijer, 457–514. New York: Springer.Google Scholar
  54. Snijders, Tom A.B., and Roel J. Bosker. 2012. Multilevel analysis: An introduction to basic and advanced multilevel modelling, 2nd Edition. London: Sage.Google Scholar
  55. Stadelmann-Steffen, Isabelle. 2012. Education policy and educational inequality—evidence from the Swiss laboratory. European Sociological Review 28:379–393.CrossRefGoogle Scholar
  56. Stegmueller, Daniel. 2013. How many countries for multilevel modeling? A comparison of frequentist and Bayesian approaches. American Journal of Political Science 57:748–761.CrossRefGoogle Scholar
  57. Stegmueller, Daniel, Peer Scheepers, Sigrid Roßteutscher and Eelke de Jong. 2012. Support for redistribution in western Europe: Assessing the role of religion. European Sociological Review 28:482–497.CrossRefGoogle Scholar
  58. Sutton, John R. 2012. Imprisonment and opportunity structures: A Bayesian hierarchical analysis. European Sociological Review 28:12–27.CrossRefGoogle Scholar
  59. Van Erp, Sara, Joris Mulder and Daniel L. Oberski. 2017. Prior sensitivity analysis in default Bayesian structural equation modeling. Psychological Methods 23:363–388.CrossRefGoogle Scholar
  60. Wilkes, Rima, Neil Guppy and Lily Farris. 2007. Right-wing parties and anti-foreigner sentiment in Europe. American Sociological Review 72:831–840.CrossRefGoogle Scholar
  61. Wooldridge, Jeffrey M. 2013. Introductory econometrics: A modern approach. Mason, OH: South-West Cengage Learning.Google Scholar
  62. Wulfgramm, M. 2014. Life satisfaction effects of unemployment in Europe: The moderating influence of labour market policy. Journal of European Social Policy 24:258–272.CrossRefGoogle Scholar

Copyright information

© Springer Fachmedien Wiesbaden GmbH, ein Teil von Springer Nature 2019

Authors and Affiliations

  • Alexander W. Schmidt-Catran
    • 1
    Email author
  • Malcolm Fairbrother
    • 2
  • Hans-Jürgen Andreß
    • 3
  1. 1.Institut für Soziologie, Lehrstuhl für Soziologie mit dem Schwerpunkt Methoden der quantitativen empirischen SozialforschungGoethe-Universität FrankfurtFrankfurt am MainGermany
  2. 2.Department of SociologyUmeå UniversityUmeåSweden
  3. 3.Fakultät für Wirtschafts- und Sozialwissenschaften, Institut für Soziologie und Sozialpsychologie, Lehrstuhl für empirische Sozial- und WirtschaftsforschungUniversität zu KölnCologneGermany

Personalised recommendations