Advertisement

Research in Higher Education

, Volume 60, Issue 3, pp 338–357 | Cite as

Should We Be Concerned About Nonresponse Bias in College Student Surveys? Evidence of Bias from a Validation Study

  • Trey StandishEmail author
  • Paul D. Umbach
Article

Abstract

This study uses college student survey data and corresponding administrative data on campus recreation facility usage, academic performance, physical education class attendance, and co-curricular participation to examine nonresponse bias in college student surveys. Within the context of the Groves (Public Opin Q 70:646–675, 2006) Alternative Cause Model, we found compelling evidence of the presence of nonresponse error observed as student characteristics related to the survey topic that also explain their response propensity. An individual’s survey response propensity has a statistically significant relationship with their actual behavior for 2 of 3 survey topics. In 11 of the 13 survey questions used to measure the survey topic behaviors, we found statistically significant differences between the respondent and nonrespondent behavioral measures. These findings hold important implications for survey researchers and those using student surveys for high-stakes accountability measures because survey summary statistics may not be generalizable to the target population.

Keywords

Nonresponse error Survey research College student surveys Nonresponse bias Validation study 

References

  1. Abraham, K. G., Helms, S., & Presser, S. (2009). How social processes distort measurement: The impact of survey nonresponse on estimates of volunteer work in the United States. American Journal of Sociology, 114(4), 1129–1165.CrossRefGoogle Scholar
  2. Adams, M. J. D., & Umbach, P. D. (2012). Nonresponse and online student evaluations of teaching: Understanding the influence of salience, fatigue, and academic environments. Research in Higher Education, 53(5), 576–591.CrossRefGoogle Scholar
  3. Aviv, A. L., Zelenski, J. M., Rallo, L., & Larsen, R. J. (2002). Who comes when: Personality differences in early and later participation in a university subject pool. Personality and Individual Differences, 33, 487–496.CrossRefGoogle Scholar
  4. Bowman, N. A. (2014). The meaning and interpretation of college student self-reporting gains. In S. Herzog & N. A. Bowman (Eds.), The validity of college student self-reported data. New Directions for Institutional Research. San Francisco: Jossey Bass.Google Scholar
  5. Clarkberg, M., Robertson, D., & Einarson, M. (2008). Engagement and student surveys: Nonresponse and implications for reporting survey data. Paper presented at the Annual Forum of the Association for Institutional Research, Seattle, WA. Ithaca, NY: Office of Institutional Research and Planning at Cornell University.Google Scholar
  6. Crawford, S. D., Couper, M. P., & Lamias, M. J. (2001). Web surveys: Perceptions of burden. Social Science Computer Review, 19(2), 146–162.CrossRefGoogle Scholar
  7. Curtin, R., Presser, S., & Singer, E. (2000). The effects of response rate changes on the index of consumer sentiment. Public Opinion Quarterly, 64(4), 413–428.CrossRefGoogle Scholar
  8. De Leeuw, E., & de Heer, W. (2002). Trends in household survey nonresponse: A longitudinal and international comparison. In R. M. Groves, D. A. Dillman, & J. L. Eltinge (Eds.), Survey nonresponse. New York: Wiley.Google Scholar
  9. Dey, E. L. (1997). Working with low survey response rates: The efficacy of weighting adjustments. Research in Higher Education, 44, 1–19.Google Scholar
  10. Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70, 646–675.CrossRefGoogle Scholar
  11. Groves, R. M., & Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72(2), 167–189.CrossRefGoogle Scholar
  12. Groves, R. M., Singer, E., & Corning, A. (2000). Leverage-saliency theory of survey participation. Public Opinion Quarterly, 64(3), 299–308.CrossRefGoogle Scholar
  13. Keeter, S., Miller, C., Kohut, A., Groves, R. M., & Presser, S. (2000). Consequences of reducing nonresponse in a national telephone survey. Public Opinion Quarterly, 64(2), 125–148.CrossRefGoogle Scholar
  14. Kuh, G. D. (2003). The national survey of student engagement: Conceptual framework and overview of psychometric properties. Bloomington, IN: Indiana University Center for Postsecondary Research.Google Scholar
  15. Merkle, D., & Edelman, M. (2002). Nonresponse in exit polls: A comprehensive analysis. In R. M. Groves, D. A. Dillman, & J. L. Eltinge (Eds.), Survey nonresponse. New York: Wiley.Google Scholar
  16. National Survey of Student Engagement. (2016). NSSE response rate FAQ. Retrieved from http://nsse.indiana.edu/pdf/Resp_Rate_FAQ.pdf.
  17. Olson, K. (2006). Survey participation, nonresponse bias, measurement error bias, and total bias. Public Opinion Quarterly, 70(5), 737–758.CrossRefGoogle Scholar
  18. Olson, K., & Kennedy, C. (2006). Examination of the relationship between nonresponse and measurement error in a validation study of alumni. Sociology Department, Faculty Publications. Paper 146.Google Scholar
  19. Pike, G. R. (2007). Adjusting for nonresponse in surveys. In John C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. 22, pp. 411–450). Dordrecht: Springer.Google Scholar
  20. Porter, S. R. (2011). Do college student surveys have any validity? The Review of Higher Education, 35(1), 45–76.CrossRefGoogle Scholar
  21. Porter, S. R., & Umbach, P. D. (2006). Student survey response rates across institutions: Why do they vary? Research in Higher Education, 47(2), 229–247.CrossRefGoogle Scholar
  22. Porter, S. R., & Whitcomb, M. E. (2005). Non-response in student surveys: The role of demographics, engagement, and personality. Research in Higher Education, 46(2), 127–152.CrossRefGoogle Scholar
  23. Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys. New York: John Wiley.CrossRefGoogle Scholar
  24. Sax, L. J., Gilmartin, S. K., & Bryant, A. N. (2003). Assessing response rates and nonresponse bias in web and paper surveys. Research in Higher Education, 44(4), 409–432.CrossRefGoogle Scholar
  25. Sax, L. J., Gilmartin, S. K., Lee, J. J., & Hagedorn, L. S. (2008). Using web surveys to reach community college students: An analysis of response rates and response bias. Community College Journal of Research and Practice, 32, 712–729.CrossRefGoogle Scholar

Copyright information

© Springer Nature B.V. 2018

Authors and Affiliations

  1. 1.SAS Institute, IncCaryUSA
  2. 2.North Carolina State UniversityRaleighUSA

Personalised recommendations