Research in Higher Education

, Volume 58, Issue 8, pp 904–921 | Cite as

The Causal Effect of Survey Mode on Students’ Evaluations of Teaching: Empirical Evidence from Three Field Experiments

  • Edgar Treischl
  • Tobias WolbringEmail author


In recent years many universities switched from paper- to online-based student evaluation of teaching (SET) without knowing the consequences for data quality. Based on a series of three consecutive field experiments—a split-half design, twin courses, and pre–post-measurements—this paper examines the effects of survey mode on SET. First, all three studies reveal marked differences in non-response between online- and paper-based SET and systematic, but small differences in the overall course ratings. On average, online SET reveal a slightly less optimistic picture of teaching quality in students’ perception. Similarly, a web survey mode does not impair the reliability of student ratings. Second, we highlight the importance of taking selection and class absenteeism into account when studying survey mode effects and also show that it is necessary and informative to survey the subgroup of no-shows when evaluating teaching. Third, we empirically demonstrate the need to account for contextual setting of the survey (in class vs. after class) and the specific type of the online survey mode (TAN vs. email). Previous research either confounded contextual setting with variation in survey mode or generalized results for a specific online mode to web surveys in general. Our findings suggest that higher response rates in email surveys can be achieved if students are given the opportunity and time to evaluate directly in class.


Students’ evaluations of teaching Online evaluation Sample selection bias Self-selection In class evaluation 


  1. Adams, M. J. D., & Umbach, P. D. (2012). Nonresponse and online student evaluations of teaching: Understanding the influence of salience, fatigue, and academic environments. Research in Higher Education, 53(5), 576–591.CrossRefGoogle Scholar
  2. Alferes, V. R. (2012). Methods of randomization in experimental design., Quantitative applications in the social sciences, 07–171 Los Angeles, CA: Sage.CrossRefGoogle Scholar
  3. Allen, I. E., & Seaman, J. (2016). Online report card: Tracking online education in the United States. Babson Survey Research Group and Quahog Research Group, LLC.Google Scholar
  4. Arulampalam, W., Naylor, R. A., & Smith, J. (2012). Am I missing something?: The effects of absence from class on student performance. Economics of Education Review, 31(4), 363–375.CrossRefGoogle Scholar
  5. Avery, R. J., Bryant, W. K., Mathios, A., Kang, H., & Bell, D. (2006). Electronic course evaluations: Does an online delivery system influence student evaluations? Journal of Economic Education, 37(1), 21–37.CrossRefGoogle Scholar
  6. Barge, S., & Gehlbach, H. (2012). Using the theory of satisficing to evaluate the quality of survey data. Research in Higher Education, 53(2), 182–200.CrossRefGoogle Scholar
  7. Barkhi, R., & Williams, P. (2010). The impact of electronic media on faculty evaluation. Assessment & Evaluation in Higher Education, 35(2), 241–262.CrossRefGoogle Scholar
  8. Beran, T. N., & Rokosh, J. L. (2009). Instructors’ perspectives on the utility of student ratings of instruction. Instructional Science, 37(2), 171–184.CrossRefGoogle Scholar
  9. Carini, R. M., Hayek, J. C., Kuh, G. D., Kennedy, J. M., & Ouimet, J. A. (2003). College student responses to web and paper surveys: Does mode matter? Research in Higher Education, 44(1), 1–19.CrossRefGoogle Scholar
  10. Champagne, M. V. (2013). Student use of mobile devices in course evaluation: A longitudinal study. Educational Research and Evaluation, 19(7), 636–646.CrossRefGoogle Scholar
  11. Cook, C., Heath, F., & Thompson, R. L. (2000). A meta-analysis of response rates in web- or internet-based surveys. Educational and Psychological Measurement, 60(6), 821–836.CrossRefGoogle Scholar
  12. Dolton, P., Marcenaro, O. D., & Navarro, L. (2003). The effective use of student time: A stochastic frontier production function case study. Economics of Education Review, 22(6), 547–560.CrossRefGoogle Scholar
  13. Dommeyer, C. J., Baum, P., Hanna, R. W., & Chapman, K. S. (2004). Gathering faculty teaching evaluations by in-class and online surveys: Their effects on response rates and evaluations. Assessment & Evaluation in Higher Education, 29(5), 611–623.CrossRefGoogle Scholar
  14. Dresel, M., & Tinsner, K. (2008). Onlineevaluation von Lehrveranstaltungen: Methodeneffekte bei der Onlineevaluation von Lehrveranstaltungen. Zeitschrift für Evaluation, 7(2), 183–211.Google Scholar
  15. Fike, D., Doyle, S., Denise, J., & Connelly, R. J. (2010). Online vs. paper evaluations of faculty: When less is just as good. The Journal of Effective Teaching, 10(2), 42–54.Google Scholar
  16. Gamliel, E., & Davidovitz, L. (2005). Online versus traditional teaching evaluation: Mode can matter. Assessment & Evaluation in Higher Education, 30(6), 581–592.CrossRefGoogle Scholar
  17. Gideon, L. (Ed.). (2012). Handbook of survey methodology for the social sciences. New York: Springer.Google Scholar
  18. Goos, M., & Salomons, A. (2016). Measuring teaching quality in higher education: Assessing selection bias in course evaluations. Research in Higher Education, doi: 10.1007/s11162-016-9429-8.Google Scholar
  19. Gravestock, P., & Gregor-Greenleaf, E. (2008). Student course evaluations: Research, models and trends. Toronto: Higher Education Quality Council of Ontario.Google Scholar
  20. Heath, N. M., Lawyer, S. R., & Rasmussen, E. B. (2007). Web-based versus paper-and-pencil course evaluations. Teaching of Psychology, 34(4), 259–261.CrossRefGoogle Scholar
  21. Imbens, G. W., & Rubin, D. B. (2015). Causal inference for statistics, social and biomedical sciences: An introduction. New York: Cambridge University Press.CrossRefGoogle Scholar
  22. Kelly, H. F., Ponton, M. K., & Rovai, A. P. (2007). A comparison of student evaluations of teaching between online and face-to-face courses. The Internet and Higher Education, 10(2), 89–101.CrossRefGoogle Scholar
  23. Kherfi, S. (2011). Whose opinion is it anyway?: Determinants of participation in student evaluation of teaching. The Journal of Economic Education, 42(1), 19–30.CrossRefGoogle Scholar
  24. Kordts-Freudinger, R., & Geithner, E. (2013). When mode does not matter: Evaluation in class versus out of class. Educational Research and Evaluation, 19(7), 605–614.CrossRefGoogle Scholar
  25. Lalla, M., & Ferrari, D. (2011). Web-based versus paper-based data collection for the evaluation of teaching activity: Empirical evidence from a case study. Assessment & Evaluation in Higher Education, 36(3), 347–365.CrossRefGoogle Scholar
  26. Layne, B. H., Decristoforo, J. R., & McGinty, D. (1999). Electronic versus traditional student ratings of instruction. Research in Higher Education, 40(2), 221–232.CrossRefGoogle Scholar
  27. Leung, D. Y. P., & Kember, D. (2005). Comparability of data gathered from evaluation questionnaires on paper and through the internet. Research in Higher Education, 46(5), 571–591.CrossRefGoogle Scholar
  28. Liegle, J. O., & McDonald, D. S. (2005). Lessons learned from online vs. paper-based computer information students’ evaluation system. Information Systems Education Journal, 37(3), 3–14.Google Scholar
  29. Marsh, H. W. (2007). students’ evaluations of university teaching: a multidimensional perspective. In R. P. Perry & J. C. Smart (Eds.), The scholarship of teaching and learning in higher education: An evidence-based perspective. New York: Springer.Google Scholar
  30. Meinefeld, W. (2010). Online-Befragungen im Kontext von Lehrevaluationen - praktisch und unzuverlässig. Kölner Zeitschrift für Soziologie und Sozialpsychologie, 62(2), 297–315.CrossRefGoogle Scholar
  31. Mesch, G. (2012). E-Mail Surveys. In L. Gideon (Ed.), Handbook of survey methodology for the Social Sciences (pp. 313–325). New York: Springer.CrossRefGoogle Scholar
  32. Morrison, K. (2013). Online and paper evaluations of courses: A literature review and case study. Educational Research and Evaluation, 19(7), 585–604.CrossRefGoogle Scholar
  33. Morrison, R. (2011). A comparison of online versus traditional student end-of-course critiques in resident courses. Assessment & Evaluation in Higher Education, 36(6), 627–641.CrossRefGoogle Scholar
  34. Mutz, R., & Daniel, H.-D. (2008). Nutzung von Lehrevaluationsdaten für die Qualitätssicherung der Evaluationsinstrumente am Beispiel der Universität Zürich. Beiträge zur Hochschulforschung, 30(2), 34–55.Google Scholar
  35. Nowell, C., Gale, L. R., & Handley, B. (2010). Assessing faculty performance using student evaluations of teaching in an uncontrolled setting. Assessment & Evaluation in Higher Education, 35(4), 463–475.CrossRefGoogle Scholar
  36. Nowell, C., Gale, L. R., & Kerkvliet, J. (2014). Non-response bias in student evaluations of teaching. International Review of Economics Education, 17, 30–38.CrossRefGoogle Scholar
  37. Nulty, D. D. (2008). The adequacy of response rates to online and paper surveys: What can be done? Assessment & Evaluation in Higher Education, 33(3), 301–314.CrossRefGoogle Scholar
  38. Perrett, J. J. (2013). Exploring graduate and undergraduate course evaluations administered on paper and online: A case study. Assessment & Evaluation in Higher Education, 38(1), 85–93.CrossRefGoogle Scholar
  39. Risquez, A., Vaughan, E., & Murphy, M. (2014). Online student evaluations of teaching: What are we sacrificing for the affordances of technology? Assessment & Evaluation in Higher Education, 40(1), 120–134.CrossRefGoogle Scholar
  40. Romer, D. (1993). Do students go to class?: Should they? Journal of Economic Perspectives, 7(3), 167–174.CrossRefGoogle Scholar
  41. Rovai, A. P., Ponton, M. K., Derrick, M. G., & Davis, J. M. (2006). Student evaluation of teaching in the virtual and traditional classrooms: A comparative analysis. The Internet and Higher Education, 9(1), 23–35.CrossRefGoogle Scholar
  42. Sax, L. J., Gilmartin, S. K., & Bryant, A. N. (2003). Assessing response rates and nonresponse bias in web and paper surveys. Research in Higher Education, 44(4), 409–432.CrossRefGoogle Scholar
  43. Schmidt, R. M. (1983). Who maximizes what? A study in student time allocation. American Economic Review, 73(2), 23–28.Google Scholar
  44. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2001). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.Google Scholar
  45. Simon, A., Zajontz, Y., & Reit, V. (2013). Lehrevaluation online oder papierbasiert?: Ein empirischer Vergleich zwischen traditionellem Fragebogen und inhaltsgleicher Online-Erhebung. Beiträge zur Hochschulforschung, 35(3), 8–26.Google Scholar
  46. Simonson, J., & Pötschke, M. (2006). Akzeptanz internetgestätzter Evaluationen an Universitäten. Zeitschrift für Evaluation, 2, 227–248.Google Scholar
  47. Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the validity of student evaluation of teaching: The state of the art. Review of Educational Research, 83(4), 598–642.CrossRefGoogle Scholar
  48. Stowell, J. R., Addison, W. E., & Smith, J. L. (2012). Comparison of online and classroom-based student evaluations of instruction. Assessment & Evaluation in Higher Education, 37(4), 465–473.CrossRefGoogle Scholar
  49. van Selm, M., & Jankowski, N. W. (2006). Conducting online surveys. Quality and Quantity, 40(3), 435–456.CrossRefGoogle Scholar
  50. Wolbring, T. (2012). Class attendance and students’ evaluations of teaching: Do no-shows bias course ratings and rankings? Evaluation Review, 36(1), 72–96.CrossRefGoogle Scholar
  51. Wolbring, T., & Treischl, E. (2016). Selection bias in students’ evaluation of teaching. Research in Higher Education, 57(1), 51–71.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.University of MannheimMannheimGermany

Personalised recommendations