Advertisement

Research in Higher Education

, Volume 57, Issue 1, pp 51–71 | Cite as

Selection Bias in Students’ Evaluation of Teaching

Causes of Student Absenteeism and Its Consequences for Course Ratings and Rankings
Article

Abstract

Systematic sampling error due to self-selection is a common topic in methodological research and a key challenge for every empirical study. Since selection bias is often not sufficiently considered as a potential flaw in research on and evaluations in higher education, the aim of this paper is to raise awareness for the topic using the case of students’ evaluations of teaching (SET). First, we describe students’ selection decisions at different points of their studies and elaborate potential biases which they might cause for SET. Then we empirically illustrate the problem and report findings from a design with two measurement points in time showing that approximately one third of the students do not attend class at the second time of measurement, when the regular SET takes place. Furthermore, the results indicate that the probability of absenteeism is influenced by course quality, students’ motivation, course topic, climate among course participants, course- and workload, and timing of the course. Although data are missing not at random, average ratings do not strongly change after adjusting for selection bias. However, we find substantial changes in rankings based on SET. We conclude from this that, at least as regards selection bias, SET are a reliable instrument to assess quality of teaching at the individual level but are not suited for the comparison of courses.

Keywords

Class attendance Missing data Students’ evaluations of teaching (SET) Sample selection bias Self-selection 

Notes

Acknowledgments

This paper has benefited from the comments of Norman Braun, Josef Brüderl, Christian Ganser, Marc Keuschnigg, Patrick Riordan, William Doyle, and two anonymous reviewers. Benedict Krauthan provided excellent research assistance.

References

  1. Abbott, A., & Tsay, A. (2000). Sequence analysis and optimal matching methods in sociology. Sociological Methods & Research, 29(1), 3–33.CrossRefGoogle Scholar
  2. Adams, M. J., & Umbach, P. D. (2012). Nonresponse and online student evaluations of teaching: Understanding the influence of salience, fatigue, and academic environments. Research in Higher Education, 53(5), 576–591.CrossRefGoogle Scholar
  3. Aina, C. (2013). Parental background and university dropout in Italy. Higher Education, 65(4), 437–456.CrossRefGoogle Scholar
  4. Allen, J., Robbins, S. B., Casillas, A., & Oh, I.-S. (2008). Third-year college retention and transfer: Effects of academic performance, motivation, and social connectedness. Research in Higher Education, 49(7), 647–664.CrossRefGoogle Scholar
  5. Arias Ortiz, E., & Dehon, C. (2013). Roads to success in the belgian French community’s higher education system: Predictors of dropout and degree completion at the Université Libre de Bruxelles. Research in Higher Education, 54(6), 693–723.CrossRefGoogle Scholar
  6. Arulampalam, W., Naylor, R. A., & Smith, J. (2012). Am I Missing Something? The effects of absence from class on student performance. Economics of Education Review, 31(4), 363–375.CrossRefGoogle Scholar
  7. Astin, A. W., & Lee, J. J. (2003). How risky are one-shot cross-sectional assessments of undergraduate students? Research in Higher Education, 44(6), 657–672.CrossRefGoogle Scholar
  8. Babad, E. (2001). Student’s course selection: Differential considerations for first and last course. Research in Higher Education, 42(4), 469–492.CrossRefGoogle Scholar
  9. Babad, E., Darley, J., & Kaplowitz, H. (1999). Developmental aspects in students’ course selection. Journal of Educational Psychology, 91, 157–168.CrossRefGoogle Scholar
  10. Babad, E., Icekson, T., & Yelinek, Y. (2008). Antecedents and correlates of course cancellation in a university “drop and add” period. Research in Higher Education, 49(4), 293–316.CrossRefGoogle Scholar
  11. Babad, E., & Tayeb, A. (2003). Experimental analysis of student’s course selection. British Journal of Educational Psychology, 73(3), 373–393.CrossRefGoogle Scholar
  12. Bahr, P. R. (2009). Educational attainment as process: Using hierarchical discrete-time event history analysis to model rate of progress. Research in Higher Education, 50(7), 691–714.CrossRefGoogle Scholar
  13. Becker, W. E., & Powers, J. R. (2001). Student performance, attrition, and class size given missing student data. Economics of Education Review, 20(4), 377–388.CrossRefGoogle Scholar
  14. Becker, W. E., & Walstad, W. B. (1990). Data loss from pretest to posttest as a sample selection problem. The Review of Economics and Statistics, 72(1), 184–188.CrossRefGoogle Scholar
  15. Berk, R. A. (2013). Top 10 flashpoints in student ratings and the evaluation of teaching. What faculty and administrators must know to protect themselves in employment decisions. Sterling: Stylus.Google Scholar
  16. Berger, U., & Schleußner, C. (2003). Are ratings of lectures confounded with students’ frequency of participation? German Journal of Educational Psychology, 17(2), 125–131.Google Scholar
  17. Bowman, N. A., & Denson, N. (2014). A missing piece of the departure puzzle: Student-institution fit and intent to persist. Research in Higher Education, 55(2), 123–142.CrossRefGoogle Scholar
  18. Bratti, M., & Staffolani, S. (2013). Student time allocation and educational production functions. Annals of Economics and Statistics, 111(112), 103–140.Google Scholar
  19. Calcagno, J. C., Crosta, P., Bailey, T., & Jenkins, D. (2007). Stepping stones to a degree: The impact of enrollment pathways and milestones on community college student outcomes. Research in Higher Education, 48(7), 775–801.CrossRefGoogle Scholar
  20. Chen, R. (2012). Institutional characteristics and college student dropout risks: A multilevel event history analysis. Research in Higher Education, 53(5), 487–505.CrossRefGoogle Scholar
  21. Chen, R., & DesJardins, S. L. (2008). Exploring the effects of financial aid on the gap in student dropout risks by income level. Research in Higher Education, 49(1), 1–18.CrossRefGoogle Scholar
  22. Coleman, J., & McKeachie, W. (1981). Effects of instructor/course evaluations on student course selection. Journal of Educational Psychology, 73, 224–226.CrossRefGoogle Scholar
  23. D’Amico, M. M., Dika, S. L., Elling, T. W., Algozzine, B., & Ginn, D. J. (2014). Early integration and other outcomes for community college transfer students. Research in Higher Education, 55(4), 370–399.CrossRefGoogle Scholar
  24. Devadoss, S., & Foltz, J. (1996). Evaluation of factors influencing student class attendance and performance. American Journal of Agricultural Economics, 78(3), 499–507.CrossRefGoogle Scholar
  25. Dolton, P., Marcenaro, O. D., & Navarro, L. (2003). The effective use of student time: A stochastic frontier production function case study. Economics of Education Review, 22(6), 547–560.CrossRefGoogle Scholar
  26. Dommeyer, C. J., Baum, P., Hanna, R. W., & Chapman, K. S. (2010). Gathering faculty teaching evaluations by in-class and online surveys: Their effects on response rates and evaluations. Assessment & Evaluation in Higher Education, 29(5), 611–623.CrossRefGoogle Scholar
  27. Douglas, S., & Sulock, J. (1995). Estimating educational production functions with correction for drops. Journal of Economic Education, 26(2), 101–112.CrossRefGoogle Scholar
  28. Elwert, F., & Winship, C. (2014). Endogenous selection bias. Annual Review of Sociology, 40.Google Scholar
  29. Enders, C. K. (2010). Applied missing data analysis. New York: Guilford Press.Google Scholar
  30. Fumasoli, T., Goastellec, G., & Kehm, B. M. (Eds.). (2015). Academic work and careers in Europe: Trends, challenges, perspectives. London: Springer.Google Scholar
  31. Gravestock, P., & Gregor-Greenleaf, E. (2008). Student course evaluations: Research, models and trends. Toronto: Higher Education Quality Council of Ontario.Google Scholar
  32. Greimel-Fuhrmann, B., & Geyer, A. (2003). Students’ evaluations of teachers and instructional quality: Analysis of relevant factors based on empirical evaluation research. Assessment & Evaluation in Higher Education, 28(3), 229–238.CrossRefGoogle Scholar
  33. Hasse, R., & Krücken, G. (2013). Competition and actorhood. A further expansion of the institutional agenda. Sociologia Internationalis, 51(2), 181–205.CrossRefGoogle Scholar
  34. Hausmann, L. R. M., Schofield, J. W., & Woods, R. L. (2007). Sense of belonging as a predictor of intentions to persist among African American and white first-year college students. Research in Higher Education, 48(7), 803–839.CrossRefGoogle Scholar
  35. Heckman, J. J. (1979). Sample selection bias as a specification error. Econometrica, 47(1), 153–161.CrossRefGoogle Scholar
  36. Herzog, S. (2005). Measuring determinants of student return vs. dropout/stopout vs. transfer: A first-to-second year analysis of new freshmen. Research in Higher Education, 46(8), 883–928.CrossRefGoogle Scholar
  37. Hochschulrektorenkonferenz, (ed.). (2010). Wegweiser 2010: Qualitätssicherung an Hochschulen. Projekt Qualitätsmanagement. Beiträge zur Hochschulpolitik 8/2010. Bonn: HRK.Google Scholar
  38. Isserstedt, W., Middendorff, E., Kandulla, M., Borchert, L., & Leszczensky, M. (2010). Die wirtschaftliche und soziale Lage der Studierenden in der Bundesrepublik Deutschland 2009. 19. Sozialerhebung des DSW durchgeführt durch HIS Hochschul-Informations-System. Bonn/Berlin: BMBF.Google Scholar
  39. Johnson, D. R., Wasserman, T. H., Yildirim, N., & Yonai, B. A. (2014). Examining the effects of stress and campus climate on the persistence of students of color and white students: An application of bean and eaton’s psychological model of retention. Research in Higher Education, 55(1), 75–100.CrossRefGoogle Scholar
  40. Johnson, I. Y. (2006). Analysis of stopout behavior at a public research university: The multi-spell discrete-time approach. Research in Higher Education, 47(8), 905–934.CrossRefGoogle Scholar
  41. Jones-White, D. R., Radcliffe, P. M., Lorenz, L. M., & Soria, K. M. (2014). Priced out? Research in Higher Education, 55(4), 329–350.CrossRefGoogle Scholar
  42. Kearney, K. A., Hopkins, R. H., Mauss, A. L., & Weisheit, R. A. (1984). Self-generated identification codes for anonymous collection of longitudinal questionnaire data. Public Opinion Quarterly, 48(1B), 370–378.CrossRefGoogle Scholar
  43. Kirby, A., & McElroy, B. (2003). The effect of attendance on grade for first year economics students in university college cork. Economic and Social Review, 34(3), 311–326.Google Scholar
  44. Lesik, S. A. (2007). Do developmental mathematics programs have a causal impact on student retention? An application of discrete-time survival and regression-discontinuity analysis. Research in Higher Education, 48(5), 583–608.CrossRefGoogle Scholar
  45. Leventhal, L., Abrami, P., & Perry, R. (1976). Do teacher rating forms reveal as much about students as about teachers? Journal of Educational Psychology, 68, 441–445.CrossRefGoogle Scholar
  46. Leventhal, L., Abrami, P., Perry, R., & Breen, L. (1975). Section selection in multi-section courses: Implications for the validation and use of teacher rating forms. Educational and Psychological Measurement, 35, 885–895.CrossRefGoogle Scholar
  47. Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing sata (2nd ed.). New York: Wiley.Google Scholar
  48. Marburger, D. R. (2001). Absenteeism and undergraduate exam performance. Journal of Economic Education, 32(2), 99–108.CrossRefGoogle Scholar
  49. Marsh, H. (2007). Students’ evaluations of university teaching: A multidimensional perspective. In P. P. Raymond & J. C. Smart (Eds.), The scholarship of teaching and learning in higher education: An evidence-based perspective (pp. 319–384). New York: Springer.CrossRefGoogle Scholar
  50. Melguizo, T. (2008). Quality matters: Assessing the impact of attending more selective institutions on college completion rates of minorities. Research in Higher Education, 49(3), 214–236.CrossRefGoogle Scholar
  51. Melguizo, T., Sanchez Torres, F., & Jaime, H. (2011). The association between financial aid availability and the college dropout rates in Colombia. Higher Education, 62(2), 231–247.CrossRefGoogle Scholar
  52. Niu, S. X., & Tienda, M. (2013). High school economic composition and college persistence. Research in Higher Education, 54(1), 30–62.CrossRefGoogle Scholar
  53. Oseguera, L., & Rhee, B. S. (2009). The influence of institutional retention climates on student persistence to degree completion: A multilevel approach. Research in Higher Education, 50(6), 546–569.CrossRefGoogle Scholar
  54. Pearl, J. (2009). Causality: Models, reasoning, and inference (2nd ed.). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  55. Reed, J. G. (1981). Dropping a college course: Factors influencing students’ withdrawal decisions. Journal of Educational Psychology, 73(3), 376–385.CrossRefGoogle Scholar
  56. Romer, D. (1993). Do students go to class? Should they? Journal of Economic Perspectives, 7(3), 167–174.CrossRefGoogle Scholar
  57. Royston, P. (2005). Multiple imputation of missing values: Update. Stata Journal, 5(2), 188–201.Google Scholar
  58. Schmidt, R. M. (1983). Who maximizes what? A study in student time allocation. American Economic Review: Papers and Proceedings, 73(2), 23–28.Google Scholar
  59. Schnell, R., Bachteler, T., & Reiher, J. (2010). Improving the use of self-generated identification codes. Evaluation Review, 34(5), 391–418.CrossRefGoogle Scholar
  60. Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the validity of student evaluation of teaching: The state of the art. Review of Educational Research, 83(4), 598–642.CrossRefGoogle Scholar
  61. Stanca, L. (2003). The effects of attendance on academic performance: Panel data evidence for introductory microeconomics. Journal of Economic Education, 37(2), 251–266.Google Scholar
  62. Taris, T. (2000). A primer in longitudinal data analysis. London: Sage.Google Scholar
  63. Tinto, V. (1975). Dropout from higher education: A theoretical synthesis of recent research. Review of Educational Research, 45(1), 89–125.CrossRefGoogle Scholar
  64. Tinto, V. (1988). Stages of student departure: Reflections on the longitudinal character of student leaving. The Journal of Higher Education, 59(4), 438–455.CrossRefGoogle Scholar
  65. Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition (2nd ed.). Chicago: University of Chicago Press.Google Scholar
  66. Titus, M. A. (2007). Detecting selection bias, using propensity score matching, and estimating treatment effects: An application to the private returns to a master’s degree. Research in Higher Education, 48(4), 487–521.CrossRefGoogle Scholar
  67. van Buuren, S. (2012). Flexible imputation of missing data. Boca Raton: CRC Press.CrossRefGoogle Scholar
  68. Wang, X. (2009). Baccalaureate attainment and college persistence of community college transfer students at four-year institutions. Research in Higher Education, 50(6), 570–588.CrossRefGoogle Scholar
  69. Wang, X., & Wickersham, K. (2014). Postsecondary Co-enrollment and baccalaureate completion: A look at both beginning 4-year college students and baccalaureate aspirants beginning at community colleges. Research in Higher Education, 55(2), 166–195.CrossRefGoogle Scholar
  70. Weiler, W. C., & Pierro, D. J. (1988). Selection bias and the analysis of persistence of part-time undergraduate students. Research in Higher Education, 29(3), 261–272.CrossRefGoogle Scholar
  71. Wilhelm, W. B. (2004). The relative influence of published teaching evaluations and other instructor attributes on course choice. Journal of Marketing Education, 26(1), 17–30.CrossRefGoogle Scholar
  72. Wolbring, T. (2012). Class attendance and students’ evaluations of teaching. Do no-shows bias course ratings and rankings? Evaluation Review, 36(1), 72–96.CrossRefGoogle Scholar
  73. Wyatt, G. (1992). Skipping class: An analysis of absenteeism among first-year college students. Teaching Sociology, 20(3), 201–207.CrossRefGoogle Scholar
  74. Yurek, L. A., Vasey, J., & Havens, D. S. (2008). The use of self-generated identification codes in longitudinal research. Evaluation Review, 32(5), 1–18.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.University of MannheimMannheimGermany
  2. 2.Institute of SociologyUniversity of MunichMunichGermany

Personalised recommendations