Should We Be Concerned About Nonresponse Bias in College Student Surveys? Evidence of Bias from a Validation Study
This study uses college student survey data and corresponding administrative data on campus recreation facility usage, academic performance, physical education class attendance, and co-curricular participation to examine nonresponse bias in college student surveys. Within the context of the Groves (Public Opin Q 70:646–675, 2006) Alternative Cause Model, we found compelling evidence of the presence of nonresponse error observed as student characteristics related to the survey topic that also explain their response propensity. An individual’s survey response propensity has a statistically significant relationship with their actual behavior for 2 of 3 survey topics. In 11 of the 13 survey questions used to measure the survey topic behaviors, we found statistically significant differences between the respondent and nonrespondent behavioral measures. These findings hold important implications for survey researchers and those using student surveys for high-stakes accountability measures because survey summary statistics may not be generalizable to the target population.
KeywordsNonresponse error Survey research College student surveys Nonresponse bias Validation study
- Bowman, N. A. (2014). The meaning and interpretation of college student self-reporting gains. In S. Herzog & N. A. Bowman (Eds.), The validity of college student self-reported data. New Directions for Institutional Research. San Francisco: Jossey Bass.Google Scholar
- Clarkberg, M., Robertson, D., & Einarson, M. (2008). Engagement and student surveys: Nonresponse and implications for reporting survey data. Paper presented at the Annual Forum of the Association for Institutional Research, Seattle, WA. Ithaca, NY: Office of Institutional Research and Planning at Cornell University.Google Scholar
- De Leeuw, E., & de Heer, W. (2002). Trends in household survey nonresponse: A longitudinal and international comparison. In R. M. Groves, D. A. Dillman, & J. L. Eltinge (Eds.), Survey nonresponse. New York: Wiley.Google Scholar
- Dey, E. L. (1997). Working with low survey response rates: The efficacy of weighting adjustments. Research in Higher Education, 44, 1–19.Google Scholar
- Kuh, G. D. (2003). The national survey of student engagement: Conceptual framework and overview of psychometric properties. Bloomington, IN: Indiana University Center for Postsecondary Research.Google Scholar
- Merkle, D., & Edelman, M. (2002). Nonresponse in exit polls: A comprehensive analysis. In R. M. Groves, D. A. Dillman, & J. L. Eltinge (Eds.), Survey nonresponse. New York: Wiley.Google Scholar
- National Survey of Student Engagement. (2016). NSSE response rate FAQ. Retrieved from http://nsse.indiana.edu/pdf/Resp_Rate_FAQ.pdf.
- Olson, K., & Kennedy, C. (2006). Examination of the relationship between nonresponse and measurement error in a validation study of alumni. Sociology Department, Faculty Publications. Paper 146.Google Scholar
- Pike, G. R. (2007). Adjusting for nonresponse in surveys. In John C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. 22, pp. 411–450). Dordrecht: Springer.Google Scholar