Skip to main content
SpringerLink
Log in

Selection Bias in Students’ Evaluation of Teaching

Causes of Student Absenteeism and Its Consequences for Course Ratings and Rankings

  • Published:
Research in Higher Education Aims and scope Submit manuscript

Abstract

Systematic sampling error due to self-selection is a common topic in methodological research and a key challenge for every empirical study. Since selection bias is often not sufficiently considered as a potential flaw in research on and evaluations in higher education, the aim of this paper is to raise awareness for the topic using the case of students’ evaluations of teaching (SET). First, we describe students’ selection decisions at different points of their studies and elaborate potential biases which they might cause for SET. Then we empirically illustrate the problem and report findings from a design with two measurement points in time showing that approximately one third of the students do not attend class at the second time of measurement, when the regular SET takes place. Furthermore, the results indicate that the probability of absenteeism is influenced by course quality, students’ motivation, course topic, climate among course participants, course- and workload, and timing of the course. Although data are missing not at random, average ratings do not strongly change after adjusting for selection bias. However, we find substantial changes in rankings based on SET. We conclude from this that, at least as regards selection bias, SET are a reliable instrument to assess quality of teaching at the individual level but are not suited for the comparison of courses.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. Exceptions are Weiler and Pierro (1988), Becker and Walstad (1990), and Titus (2007).

  2. These results are based on in-course evaluations and, thus, only encompass ratings of students which were present in class at the time of the evaluation. If our main hypothesis of selective, quality-induced migration holds, then average course ratings reported in Fig. 1 are positively biased and more strongly so, if dropout is high. Thus, these data very likely underestimate the actual relationship between both variables but at least give a lower bound of the strength of the association.

  3. This clearly illustrates why graduate surveys are an imperfect instrument for measuring the quality of teaching and why one should be careful to generalize findings to whole student cohorts: If the group of graduates systematically differs from dropouts in terms of unobserved heterogeneity (e.g. motivation, satisfaction, success) which correlates with the assessment of the study program and teaching quality, then the results of graduation surveys will be positively biased. A direct implication from these considerations is that one should not restrict the sample on graduates but better survey cohorts of first-year students and collect longitudinal data on their study path, attainment, and achievement

  4. Bahr mentioned: ,,persistence, enrollment inconsistency, completed credit hours, course credit load, course completion rate, and delay of first enrollment in core subjects” (Bahr 2009, p. 692).

  5. In 2009, 87 % of the students in Germany were financially supported by their parents, 65 % worked besides their studies, and another 29 % received state funding (Isserstedt et al. 2010, p. 193).

  6. At this point we would like to thank both the lecturers as well as students who participated. This study could not have been realized without their support and cooperation.

  7. The self-generated identification code was based on person-specific and time constant information such as the first two letters of mother and father first name, number of older sisters and brothers, etc. For general information on identification codes see Kearney et al. (1984) and Yurek et al. (2008).

  8. An alternative approach to this matching task is the use of sequence analysis. Thereby, one first calculates similarity measures and then matches observations for which the similarity measure exceeds a certain threshold (for introductions into sequence analysis see Abbott and Tsay 2000; Taris 2000; for an application using identification codes see Schnell et al. 2010).

  9. For the self-generated identification code the missing data problem seems neglectable. From an overall 2.263 single observations only 40 gave no personal information at all.

  10. We used the Stata ados ice (Royston 2005) and mim and included the outcome variable, all independent variables of our analyses, and some additional variables measured at t = 1 as predictors. Ordinal variables were estimated with ordinal logistic regressions, all other variables with simple linear regressions with 100 imputations. To test for the robustness of our approach, we additionally imputed the missing data using multivariate normal regression (Stata command mi impute mvn). Results did not change. Moreover, we compared our results with estimates based on listwise deletion (N = 1056). Regression coefficients and p-values changed, but the substantial findings remained untouched.

References

  • Abbott, A., & Tsay, A. (2000). Sequence analysis and optimal matching methods in sociology. Sociological Methods & Research, 29(1), 3–33.

    Article  Google Scholar 

  • Adams, M. J., & Umbach, P. D. (2012). Nonresponse and online student evaluations of teaching: Understanding the influence of salience, fatigue, and academic environments. Research in Higher Education, 53(5), 576–591.

    Article  Google Scholar 

  • Aina, C. (2013). Parental background and university dropout in Italy. Higher Education, 65(4), 437–456.

    Article  Google Scholar 

  • Allen, J., Robbins, S. B., Casillas, A., & Oh, I.-S. (2008). Third-year college retention and transfer: Effects of academic performance, motivation, and social connectedness. Research in Higher Education, 49(7), 647–664.

    Article  Google Scholar 

  • Arias Ortiz, E., & Dehon, C. (2013). Roads to success in the belgian French community’s higher education system: Predictors of dropout and degree completion at the Université Libre de Bruxelles. Research in Higher Education, 54(6), 693–723.

    Article  Google Scholar 

  • Arulampalam, W., Naylor, R. A., & Smith, J. (2012). Am I Missing Something? The effects of absence from class on student performance. Economics of Education Review, 31(4), 363–375.

    Article  Google Scholar 

  • Astin, A. W., & Lee, J. J. (2003). How risky are one-shot cross-sectional assessments of undergraduate students? Research in Higher Education, 44(6), 657–672.

    Article  Google Scholar 

  • Babad, E. (2001). Student’s course selection: Differential considerations for first and last course. Research in Higher Education, 42(4), 469–492.

    Article  Google Scholar 

  • Babad, E., Darley, J., & Kaplowitz, H. (1999). Developmental aspects in students’ course selection. Journal of Educational Psychology, 91, 157–168.

    Article  Google Scholar 

  • Babad, E., Icekson, T., & Yelinek, Y. (2008). Antecedents and correlates of course cancellation in a university “drop and add” period. Research in Higher Education, 49(4), 293–316.

    Article  Google Scholar 

  • Babad, E., & Tayeb, A. (2003). Experimental analysis of student’s course selection. British Journal of Educational Psychology, 73(3), 373–393.

    Article  Google Scholar 

  • Bahr, P. R. (2009). Educational attainment as process: Using hierarchical discrete-time event history analysis to model rate of progress. Research in Higher Education, 50(7), 691–714.

    Article  Google Scholar 

  • Becker, W. E., & Powers, J. R. (2001). Student performance, attrition, and class size given missing student data. Economics of Education Review, 20(4), 377–388.

    Article  Google Scholar 

  • Becker, W. E., & Walstad, W. B. (1990). Data loss from pretest to posttest as a sample selection problem. The Review of Economics and Statistics, 72(1), 184–188.

    Article  Google Scholar 

  • Berk, R. A. (2013). Top 10 flashpoints in student ratings and the evaluation of teaching. What faculty and administrators must know to protect themselves in employment decisions. Sterling: Stylus.

    Google Scholar 

  • Berger, U., & Schleußner, C. (2003). Are ratings of lectures confounded with students’ frequency of participation? German Journal of Educational Psychology, 17(2), 125–131.

    Google Scholar 

  • Bowman, N. A., & Denson, N. (2014). A missing piece of the departure puzzle: Student-institution fit and intent to persist. Research in Higher Education, 55(2), 123–142.

    Article  Google Scholar 

  • Bratti, M., & Staffolani, S. (2013). Student time allocation and educational production functions. Annals of Economics and Statistics, 111(112), 103–140.

    Google Scholar 

  • Calcagno, J. C., Crosta, P., Bailey, T., & Jenkins, D. (2007). Stepping stones to a degree: The impact of enrollment pathways and milestones on community college student outcomes. Research in Higher Education, 48(7), 775–801.

    Article  Google Scholar 

  • Chen, R. (2012). Institutional characteristics and college student dropout risks: A multilevel event history analysis. Research in Higher Education, 53(5), 487–505.

    Article  Google Scholar 

  • Chen, R., & DesJardins, S. L. (2008). Exploring the effects of financial aid on the gap in student dropout risks by income level. Research in Higher Education, 49(1), 1–18.

    Article  Google Scholar 

  • Coleman, J., & McKeachie, W. (1981). Effects of instructor/course evaluations on student course selection. Journal of Educational Psychology, 73, 224–226.

    Article  Google Scholar 

  • D’Amico, M. M., Dika, S. L., Elling, T. W., Algozzine, B., & Ginn, D. J. (2014). Early integration and other outcomes for community college transfer students. Research in Higher Education, 55(4), 370–399.

    Article  Google Scholar 

  • Devadoss, S., & Foltz, J. (1996). Evaluation of factors influencing student class attendance and performance. American Journal of Agricultural Economics, 78(3), 499–507.

    Article  Google Scholar 

  • Dolton, P., Marcenaro, O. D., & Navarro, L. (2003). The effective use of student time: A stochastic frontier production function case study. Economics of Education Review, 22(6), 547–560.

    Article  Google Scholar 

  • Dommeyer, C. J., Baum, P., Hanna, R. W., & Chapman, K. S. (2010). Gathering faculty teaching evaluations by in-class and online surveys: Their effects on response rates and evaluations. Assessment & Evaluation in Higher Education, 29(5), 611–623.

    Article  Google Scholar 

  • Douglas, S., & Sulock, J. (1995). Estimating educational production functions with correction for drops. Journal of Economic Education, 26(2), 101–112.

    Article  Google Scholar 

  • Elwert, F., & Winship, C. (2014). Endogenous selection bias. Annual Review of Sociology, 40.

  • Enders, C. K. (2010). Applied missing data analysis. New York: Guilford Press.

    Google Scholar 

  • Fumasoli, T., Goastellec, G., & Kehm, B. M. (Eds.). (2015). Academic work and careers in Europe: Trends, challenges, perspectives. London: Springer.

    Google Scholar 

  • Gravestock, P., & Gregor-Greenleaf, E. (2008). Student course evaluations: Research, models and trends. Toronto: Higher Education Quality Council of Ontario.

    Google Scholar 

  • Greimel-Fuhrmann, B., & Geyer, A. (2003). Students’ evaluations of teachers and instructional quality: Analysis of relevant factors based on empirical evaluation research. Assessment & Evaluation in Higher Education, 28(3), 229–238.

    Article  Google Scholar 

  • Hasse, R., & Krücken, G. (2013). Competition and actorhood. A further expansion of the institutional agenda. Sociologia Internationalis, 51(2), 181–205.

    Article  Google Scholar 

  • Hausmann, L. R. M., Schofield, J. W., & Woods, R. L. (2007). Sense of belonging as a predictor of intentions to persist among African American and white first-year college students. Research in Higher Education, 48(7), 803–839.

    Article  Google Scholar 

  • Heckman, J. J. (1979). Sample selection bias as a specification error. Econometrica, 47(1), 153–161.

    Article  Google Scholar 

  • Herzog, S. (2005). Measuring determinants of student return vs. dropout/stopout vs. transfer: A first-to-second year analysis of new freshmen. Research in Higher Education, 46(8), 883–928.

    Article  Google Scholar 

  • Hochschulrektorenkonferenz, (ed.). (2010). Wegweiser 2010: Qualitätssicherung an Hochschulen. Projekt Qualitätsmanagement. Beiträge zur Hochschulpolitik 8/2010. Bonn: HRK.

  • Isserstedt, W., Middendorff, E., Kandulla, M., Borchert, L., & Leszczensky, M. (2010). Die wirtschaftliche und soziale Lage der Studierenden in der Bundesrepublik Deutschland 2009. 19. Sozialerhebung des DSW durchgeführt durch HIS Hochschul-Informations-System. Bonn/Berlin: BMBF.

  • Johnson, D. R., Wasserman, T. H., Yildirim, N., & Yonai, B. A. (2014). Examining the effects of stress and campus climate on the persistence of students of color and white students: An application of bean and eaton’s psychological model of retention. Research in Higher Education, 55(1), 75–100.

    Article  Google Scholar 

  • Johnson, I. Y. (2006). Analysis of stopout behavior at a public research university: The multi-spell discrete-time approach. Research in Higher Education, 47(8), 905–934.

    Article  Google Scholar 

  • Jones-White, D. R., Radcliffe, P. M., Lorenz, L. M., & Soria, K. M. (2014). Priced out? Research in Higher Education, 55(4), 329–350.

    Article  Google Scholar 

  • Kearney, K. A., Hopkins, R. H., Mauss, A. L., & Weisheit, R. A. (1984). Self-generated identification codes for anonymous collection of longitudinal questionnaire data. Public Opinion Quarterly, 48(1B), 370–378.

    Article  Google Scholar 

  • Kirby, A., & McElroy, B. (2003). The effect of attendance on grade for first year economics students in university college cork. Economic and Social Review, 34(3), 311–326.

    Google Scholar 

  • Lesik, S. A. (2007). Do developmental mathematics programs have a causal impact on student retention? An application of discrete-time survival and regression-discontinuity analysis. Research in Higher Education, 48(5), 583–608.

    Article  Google Scholar 

  • Leventhal, L., Abrami, P., & Perry, R. (1976). Do teacher rating forms reveal as much about students as about teachers? Journal of Educational Psychology, 68, 441–445.

    Article  Google Scholar 

  • Leventhal, L., Abrami, P., Perry, R., & Breen, L. (1975). Section selection in multi-section courses: Implications for the validation and use of teacher rating forms. Educational and Psychological Measurement, 35, 885–895.

    Article  Google Scholar 

  • Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing sata (2nd ed.). New York: Wiley.

    Google Scholar 

  • Marburger, D. R. (2001). Absenteeism and undergraduate exam performance. Journal of Economic Education, 32(2), 99–108.

    Article  Google Scholar 

  • Marsh, H. (2007). Students’ evaluations of university teaching: A multidimensional perspective. In P. P. Raymond & J. C. Smart (Eds.), The scholarship of teaching and learning in higher education: An evidence-based perspective (pp. 319–384). New York: Springer.

    Chapter  Google Scholar 

  • Melguizo, T. (2008). Quality matters: Assessing the impact of attending more selective institutions on college completion rates of minorities. Research in Higher Education, 49(3), 214–236.

    Article  Google Scholar 

  • Melguizo, T., Sanchez Torres, F., & Jaime, H. (2011). The association between financial aid availability and the college dropout rates in Colombia. Higher Education, 62(2), 231–247.

    Article  Google Scholar 

  • Niu, S. X., & Tienda, M. (2013). High school economic composition and college persistence. Research in Higher Education, 54(1), 30–62.

    Article  Google Scholar 

  • Oseguera, L., & Rhee, B. S. (2009). The influence of institutional retention climates on student persistence to degree completion: A multilevel approach. Research in Higher Education, 50(6), 546–569.

    Article  Google Scholar 

  • Pearl, J. (2009). Causality: Models, reasoning, and inference (2nd ed.). Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Reed, J. G. (1981). Dropping a college course: Factors influencing students’ withdrawal decisions. Journal of Educational Psychology, 73(3), 376–385.

    Article  Google Scholar 

  • Romer, D. (1993). Do students go to class? Should they? Journal of Economic Perspectives, 7(3), 167–174.

    Article  Google Scholar 

  • Royston, P. (2005). Multiple imputation of missing values: Update. Stata Journal, 5(2), 188–201.

    Google Scholar 

  • Schmidt, R. M. (1983). Who maximizes what? A study in student time allocation. American Economic Review: Papers and Proceedings, 73(2), 23–28.

    Google Scholar 

  • Schnell, R., Bachteler, T., & Reiher, J. (2010). Improving the use of self-generated identification codes. Evaluation Review, 34(5), 391–418.

    Article  Google Scholar 

  • Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the validity of student evaluation of teaching: The state of the art. Review of Educational Research, 83(4), 598–642.

    Article  Google Scholar 

  • Stanca, L. (2003). The effects of attendance on academic performance: Panel data evidence for introductory microeconomics. Journal of Economic Education, 37(2), 251–266.

    Google Scholar 

  • Taris, T. (2000). A primer in longitudinal data analysis. London: Sage.

    Google Scholar 

  • Tinto, V. (1975). Dropout from higher education: A theoretical synthesis of recent research. Review of Educational Research, 45(1), 89–125.

    Article  Google Scholar 

  • Tinto, V. (1988). Stages of student departure: Reflections on the longitudinal character of student leaving. The Journal of Higher Education, 59(4), 438–455.

    Article  Google Scholar 

  • Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition (2nd ed.). Chicago: University of Chicago Press.

    Google Scholar 

  • Titus, M. A. (2007). Detecting selection bias, using propensity score matching, and estimating treatment effects: An application to the private returns to a master’s degree. Research in Higher Education, 48(4), 487–521.

    Article  Google Scholar 

  • van Buuren, S. (2012). Flexible imputation of missing data. Boca Raton: CRC Press.

    Book  Google Scholar 

  • Wang, X. (2009). Baccalaureate attainment and college persistence of community college transfer students at four-year institutions. Research in Higher Education, 50(6), 570–588.

    Article  Google Scholar 

  • Wang, X., & Wickersham, K. (2014). Postsecondary Co-enrollment and baccalaureate completion: A look at both beginning 4-year college students and baccalaureate aspirants beginning at community colleges. Research in Higher Education, 55(2), 166–195.

    Article  Google Scholar 

  • Weiler, W. C., & Pierro, D. J. (1988). Selection bias and the analysis of persistence of part-time undergraduate students. Research in Higher Education, 29(3), 261–272.

    Article  Google Scholar 

  • Wilhelm, W. B. (2004). The relative influence of published teaching evaluations and other instructor attributes on course choice. Journal of Marketing Education, 26(1), 17–30.

    Article  Google Scholar 

  • Wolbring, T. (2012). Class attendance and students’ evaluations of teaching. Do no-shows bias course ratings and rankings? Evaluation Review, 36(1), 72–96.

    Article  Google Scholar 

  • Wyatt, G. (1992). Skipping class: An analysis of absenteeism among first-year college students. Teaching Sociology, 20(3), 201–207.

    Article  Google Scholar 

  • Yurek, L. A., Vasey, J., & Havens, D. S. (2008). The use of self-generated identification codes in longitudinal research. Evaluation Review, 32(5), 1–18.

    Article  Google Scholar 

Download references

Acknowledgments

This paper has benefited from the comments of Norman Braun, Josef Brüderl, Christian Ganser, Marc Keuschnigg, Patrick Riordan, William Doyle, and two anonymous reviewers. Benedict Krauthan provided excellent research assistance.

Author information

Authors and Affiliations

  1. University of Mannheim, A5, 6, 68131, Mannheim, Germany

    Tobias Wolbring

  2. Institute of Sociology, University of Munich, Konradstr. 6, 80801, Munich, Germany

    Edgar Treischl

Authors
  1. Tobias Wolbring
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Edgar Treischl
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tobias Wolbring.

Appendix

Appendix

See Table 3.

Table 3 Descriptive statistics for students present at t = 1
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wolbring, T., Treischl, E. Selection Bias in Students’ Evaluation of Teaching. Res High Educ 57, 51–71 (2016). https://doi.org/10.1007/s11162-015-9378-7

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11162-015-9378-7

Keywords

Search

Navigation