Skip to main content

Advertisement

SpringerLink
Log in

Nonresponse and Online Student Evaluations of Teaching: Understanding the Influence of Salience, Fatigue, and Academic Environments

  • Published:
Research in Higher Education Aims and scope Submit manuscript

Abstract

Technological advances have enabled institutions of higher education to administer course evaluations online, forgoing the traditional paper-and-pencil methods. Consequently, many of these institutions suffer from low response rates, but little research is available on this topic. To increase understanding about course evaluation participation in the online environment, this study examined over 22,000 undergraduates to whom the university administered about 135,000 evaluations. Multilevel models were constructed to analyze the data, and several variables emerged as significant predictors of participation. The results were mostly consistent with previous research and aligned with theories of survey nonresponse. However, the integration of uncommon variables provided new perspectives about course evaluations in particular. Implications for research and practical applications for institutions are also addressed, including ways to combat survey fatigue, increase the salience of the survey, and increase participation in online course evaluations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. HLM equations were too lengthy to insert in this article. These are available by request from the first author.

  2. The full models are too large to present in this article, but are available by request from the first author.

References

  • Avery, R. J., Bryant, W. K., Mathios, A., Kang, H., & Bell, D. (2006). Electronic SETs: Does an online delivery system influence student evaluations? Journal of Economic Education, 37, 21–37.

    Article  Google Scholar 

  • Blau, P. (1964). Exchange and power in social life. New York: Wiley.

    Google Scholar 

  • Brandenburg, G. C., & Remmers, H. H. (1927). The Purdue Rating Scale for instructors. Educational Administration and Supervision, 13, 399–406.

    Google Scholar 

  • Clarksberg, M., Robertson, D., & Einarson, M. (2008). Engagement and student surveys: Nonresponse and implications for reporting survey data. Paper presented at the 48th Annual Forum of the Association for Institutional Research, Seattle, WA.

  • Cohen, P. A. (1981). Student ratings of instruction and student achievement: A meta-analysis of multisection validity studies. Review of Educational Research, 51(3), 281–309.

    Google Scholar 

  • Costin, F. (1978). Do student ratings of college teachers predict student achievement? Teaching of Psychology, 5(2), 86–88.

    Article  Google Scholar 

  • Crumbley, D. L., & Reichelt, K. J. (2009). Teaching effectiveness, impression management, and dysfunctional behavior: Student evaluation of teaching control data. Quality Assurance in Education, 17(4), 377–392.

    Article  Google Scholar 

  • Davis, M., Hirschberg, J., Lye, J. N., & Johnston, C. G. (2007). Systematic influences on teaching evaluations: The case for caution. Australian Economic Papers, 46(1), 18–38.

    Article  Google Scholar 

  • Dey, E. L. (1997). Working with low survey response rates: The efficacy of weighting adjustments. Research in Higher Education, 38(2), 215–227.

    Article  Google Scholar 

  • Dillman, D. A., Eltinge, J. L., Groves, R. M., & Little, R. J. A. (2002). Survey nonresponse in design, data collection, and analysis. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 3–26). New York: Wiley.

    Google Scholar 

  • Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009). Internet, mail, and mixed-mode surveys: The tailored design method. San Francisco: Jossey-Bass.

    Google Scholar 

  • Dommeyer, C. J., Baum, P., Hanna, R. W., & Chapman, K. S. (2004). Gathering faculty teaching evaluations by in-class and online surveys: Their effects on response rates and evaluations. Assessment and Evaluation in Higher Education, 29(5), 611–625.

    Article  Google Scholar 

  • Etzioni, A. (1975). A comparative analysis of complex organizations: On power, involvement, and their correlates. New York: The Free Press.

    Google Scholar 

  • Fidelman, C. G. (2007). Course evaluation surveys: In-class paper surveys versus voluntary online surveys. Doctoral dissertation (UMI no. 3301790), Boston College.

  • Goyder, J., Warriner, K., & Miller, S. (2002). Evaluating socio-economic status (SES) bias in survey nonresponse. Journal of Official Statistics, 18(1), 1–11.

    Google Scholar 

  • Groves, R. M. (1989). Survey errors and survey costs. New York: Wiley.

    Book  Google Scholar 

  • Groves, R. M., & Couper, M. P. (1998). Nonresponse in household interview surveys. New York: Wiley.

    Google Scholar 

  • Groves, R. M., Couper, M., Presser, S., Singer, E., Tourangeau, R., Piani Acosta, G., et al. (2006). Experiments in producing nonresponse bias. Public Opinion Quarterly, 70(5), 720–736.

    Article  Google Scholar 

  • Groves, R. M., Dillman, D. A., Eltinge, J. L., & Little, R. J. A. (2002). Survey nonresponse. New York: Wiley.

    Google Scholar 

  • Groves, R. M., Fowler, F. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2004a). Survey methodology. Hoboken, NJ: Wiley.

    Google Scholar 

  • Groves, R. M., Fowler, F. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey methodology (2nd ed.). Hoboken, NJ: Wiley.

    Google Scholar 

  • Groves, R. M., & Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72(2), 167–189.

    Article  Google Scholar 

  • Groves, R. M., Presser, S., & Dipko, S. (2004b). The role of topic interest in survey participation decisions. Public Opinion Quarterly, 68(1), 2–31.

    Article  Google Scholar 

  • Guthrie, E. R. (1954). The evaluation of teaching: A progress report. Seattle: University of Washington.

    Google Scholar 

  • Heberlein, T. A., & Baumgartner, R. (1978). Factors affecting response rates to mailed questionnaires: A quantitative analysis of the published literature. American Sociological Review, 43(4), 447–462.

    Article  Google Scholar 

  • Ho, D. E., & Shapiro, T. H. (2008). Evaluating course evaluations: An empirical analysis of a quasi-experiment at the Stanford Law School, 2000-2007. Journal of Legal Education, 58(3), 388–412.

    Google Scholar 

  • Hofstede, G. (1981). Culture and organizations. International Studies of Management and Organization, 10(4), 15–41.

    Google Scholar 

  • Holland, J. L. (1966). The psychology of vocational choice. Waltham, MA: Blaisdell.

    Google Scholar 

  • Holland, J. L. (1973). Making vocational choice: A theory of careers. Englewood Cliffs, NJ: Prentice Hall.

    Google Scholar 

  • Holland, J. L. (1997). Making vocational choice: A theory of vocational personalities and work environments. Odessa, FL: Psychological Assessment Resources.

    Google Scholar 

  • Isely, P., & Singh, H. (2005). Do higher grades lead to favorable student evaluations? Journal of Economic Education, 36(1), 29–42.

    Article  Google Scholar 

  • Johnson, T. D. (2003). Online student ratings: Will students respond? In D. L. Sorenson & T. D. Johnson (Eds.), Online student ratings of instruction: Vol. 96. New directions for teaching and learning (pp. 49–59). Hoboken, NJ: Wiley.

    Google Scholar 

  • Johnson, T. P., O’Rourke, D., Burris, J., & Owens, L. (2002). Culture and survey nonresponse. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 55–70). New York: Wiley.

    Google Scholar 

  • Jones, C. R. (2009). Nonresponse bias in online SETs. Doctoral dissertation (UMI no. 3386938), James Madison University.

  • Kucsera, J. V., & Zimmaro, D. M. (2008). Electronic course instructor survey (eCIS) report. Austin, TX: Division of Instructional Innovation and Assessment, University of Texas at Austin.

  • Layne, B. H., DeCristoforo, J. R., & McGinty, D. (1999). Electronic versus traditional student ratings of instruction. Research in Higher Education, 40(2), 221–232.

    Article  Google Scholar 

  • Lindner, J. R., Murphy, T. H., & Briers, G. H. (2001). Handling nonresponse in social science research. Journal of Agricultural Education, 42(4), 43–53.

    Article  Google Scholar 

  • Marcus, B., & Schutz, A. (2005). Who are the people reluctant to participate in research? Personality correlates of four different types of nonresponse as inferred from self- and observer ratings. Journal of Personality, 73(4), 1–26.

    Article  Google Scholar 

  • Marsh, H. W. (2007). Students’ evaluations of university teaching: Dimensionality, reliability, validity, potential biases and usefulness. In R. P. Perry & J. C. Smart (Eds.), The scholarship of teaching and learning in higher education: An evidence-based perspective (pp. 319–383). Dordrecht: Springer.

    Chapter  Google Scholar 

  • Marsh, H. W., & Roche, L. A. (1997). Making students’ evaluations of teaching effectiveness effective. American Psychologist, 52, 1187–1197.

    Article  Google Scholar 

  • McGourty, J., Scoles, K., & Thorpe, S. (2002a, June). Web-based student evaluation of instruction: Promises and pitfalls. Paper presented at the 42nd Annual Forum of the Association for Institutional Research, Toronto, ON, Canada.

  • McGourty, J., Scoles, K., & Thorpe, S. (2002b, November). Web-based course evaluation: Comparing the experience at two universities. Paper presented at the 32nd ASEE/IEEE Frontiers in Education Conference, Boston, MA.

  • McKeachie, W. J. (1969). Student ratings of faculty. American Association of University Professors Bulletin, 55, 439–444.

    Google Scholar 

  • McKeachie, W. J. (1979). Student ratings of faculty: A reprise. Academe, 65, 384–397.

    Article  Google Scholar 

  • Moore, D. L., & Tarnai, J. (2002). Evaluating nonresponse error in mail surveys. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 197–212). New York: Wiley.

    Google Scholar 

  • Porter, S. R. (2004a). Pros and cons of paper and electronic surveys. In S. R. Porter (Ed.), Overcoming survey research problems: Vol. 121. New directions for institutional research (pp. 91–98). San Francisco, CA: Jossey-Bass.

    Google Scholar 

  • Porter, S. R. (2004b). Raising response rates: What works? In S. R. Porter (Ed.), Overcoming survey research problems: Vol. 121. New directions for institutional research (pp. 5–22). San Francisco, CA: Jossey-Bass.

    Google Scholar 

  • Porter, S. R., & Umbach, P. D. (2006). Student survey response rates across institutions: Why do they vary? Research in Higher Education, 47(2), 229–247.

    Article  Google Scholar 

  • Porter, S. R., & Whitcomb, M. E. (2005). Non-response in student surveys: The role of demographics, engagement, and personality. Research in Higher Education, 46(2), 127–152.

    Article  Google Scholar 

  • Porter, S. R., & Whitcomb, M. E. (2007). Mixed-mode contacts in web surveys: Paper is not necessarily better. Public Opinion Quarterly, 71(4), 635–648.

    Article  Google Scholar 

  • Porter, S. R., Whitcomb, M. E., & Weitzer, W. H. (2004). Multiple surveys of students and survey fatigue. In S. R. Porter (Ed.), Overcoming survey research problems: Vol. 121. New directions for institutional research (pp. 63–74). San Francisco, CA: Jossey-Bass.

    Google Scholar 

  • Remedios, R., & Lieberman, D. A. (2008). I liked your course because you taught me well: The influence of grades, workload, expectations and goals on students’ evaluations of teaching. British Educational Research Journal, 34(1), 91–115.

    Article  Google Scholar 

  • Rosen, D., Holmberg, K., & Holland, J. L. (1997). The educational opportunities finder. Odessa, FL: Psychological Assessment Resources.

    Google Scholar 

  • Rubin, D. B., & Zanutto, E. (2002). Using matched substitutes to adjust for nonignorable nonresponse through multiple imputations. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 389–402). New York: Wiley.

    Google Scholar 

  • Sax, L. J., Gilmartin, S. K., & Bryant, A. N. (2003). Assessing response rates and nonresponse bias in web and paper surveys. Research in Higher Education, 44(4), 409–432.

    Article  Google Scholar 

  • Sax, L. J., Gilmartin, S. K., Lee, J. J., & Hagedorn, L. S. (2008). Using web surveys to reach community college students: An analysis of response rates and response bias. Community College Journal of Research & Practice, 32(9), 712–729.

    Article  Google Scholar 

  • Smart, J. C., Feldman, K. A., & Ethington, C. A. (2000). Academic disciplines: Holland’s theory and the study of college students and faculty. Nashville, TN: Vanderbilt University Press.

    Google Scholar 

  • Thorpe, S. W. (2002). Online student evaluation of instruction: An investigation of non-response bias. Paper presented at the 42nd Annual Forum for the Association for Institutional Research, Toronto, ON, Canada.

  • Umbach, P. D. (2005). Getting back to the basics of survey research. In P. D. Umbach (Ed.), Survey research: Emerging issues: Vol. 127. New directions for institutional research (pp. 91–100). San Francisco, CA: Jossey-Bass.

    Google Scholar 

  • Vehovar, V., Batagelj, Z., Manfreda, K. L., & Zaletel, M. (2002). Nonresponse in web surveys. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 229–242). New York: Wiley.

    Google Scholar 

  • Yu, C. H., Jannasch-Pennell, A., DiGangi, S., Kim, C., & Andrews, S. (2007). Data visualization and mining for survey responses. Practical Assessment, Research & Evaluation, 12(19), 1–12.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Leadership, Policy, and Adult and Higher Education, North Carolina State University, 530 Poe Hall, Campus Box 7801, Raleigh, NC, 27695-7801, USA

    Meredith J. D. Adams

  2. Department of Leadership, Policy, and Adult and Higher Education, North Carolina State University, 300 Poe Hall, Campus Box 7801, Raleigh, NC, 27695-7801, USA

    Paul D. Umbach

Authors
  1. Meredith J. D. Adams
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paul D. Umbach
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Meredith J. D. Adams.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Adams, M.J.D., Umbach, P.D. Nonresponse and Online Student Evaluations of Teaching: Understanding the Influence of Salience, Fatigue, and Academic Environments. Res High Educ 53, 576–591 (2012). https://doi.org/10.1007/s11162-011-9240-5

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11162-011-9240-5

Keywords

Search

Navigation