Research in Higher Education

, Volume 44, Issue 4, pp 409–432 | Cite as

Assessing Response Rates and Nonresponse Bias in Web and Paper Surveys

  • Linda J. Sax
  • Shannon K. Gilmartin
  • Alyssa N. Bryant
Article

Abstract

Using data collected as part of the second pilot administration of Your First College Year (YFCY), a national survey of first-year college students, this study was designed to examine both response rates and nonresponse bias across four survey administration groups: paper-only, paper with web option, web-only with response incentive, and web-only without response incentive. Findings indicate that response rates vary by mode of administration. Moreover, predictors of response differed by administration group. Results are discussed in light of the recent surge of interest in online survey research.

nonresponse bias response rates online survey research college student surveys 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

REFERENCES

  1. Antons, C. M., Dilla, B. L., and Fultz, M. L. (1997). Assessing student attitudes: Computer vs. pencil. Paper presented at the 37th Annual Forum of the Association for Institutional Research, Orlando, FL, May 18–21, 1997.Google Scholar
  2. Astin, A. W. (1993). An empirical typology of college students. J. Coll. Stud. Dev. 34: 36–46.Google Scholar
  3. Bradburn, N. M. (1992). Presidential address: A response to the non-response problem. Public Opin. Q. 56: 391–398.Google Scholar
  4. Carini, R. M., Hayek, J. C., Kuh, G. D., Kennedy, J. M., and Ouimet, J. A. (2001). College student responses to web and paper surveys: Does mode matter? Paper presented at the 41st Annual Forum of the Association for Institutional Research. Long Beach, CA, June 4, 2001.Google Scholar
  5. Caslyn, R. J., and Winter, J. P. (1999). Understanding and controlling response bias in need assessment studies. Eval. Rev. 23: 399–417.Google Scholar
  6. Clayton, R. L., and Werking, G. S. (1998). Business surveys of the future: The World Wide Web as a data collection method. In: Couper, M. P., Baker, R. P., Bethlehem, J., Clark, Z. F., Martin, J., Nicholls II, W. L., and O'Reilly, J. M. (eds.), Computer Assisted Survey Information Collection, John Wiley & Sons, New York, pp. 543–562.Google Scholar
  7. Couper, M. P. (2000). Web surveys: A review of issues and approaches. Public Opin. Q. 64: 464–494.Google Scholar
  8. Creswell, J. W. (1994). Research Design: Qualitative and Quantitative Approaches, Sage Publications, Thousand Oaks.Google Scholar
  9. de Leeuw, E., and Heer, W. (2002). Trends in household survey nonresponse: A longitudinal and international comparison. In: Groves, R. M., Dillman, D. A., Eltinge, J. L., and Little, R. J. A. (eds.), Survey Nonresponse, John Wiley & Sons, New York, pp. 41–54.Google Scholar
  10. Dey, E. L. (1997). Working with low survey response rates: The efficacy of weighing adjustments. Res. Higher Educ. 38: 215–227.Google Scholar
  11. Dillman, D. A. (1991). The design and administration of mail surveys. Annu. Rev. Sociol. 17: 225–249.Google Scholar
  12. Dillman, D. A. (2000). Mail and Internet Surveys: The Tailored Design Method (2nd Ed.), John Wiley & Sons, New York.Google Scholar
  13. Fraenkel, J. R., and Wallen, N. E. (1993). How to Design and Evaluate Research in Education (2nd Ed.), McGraw Hill, New York.Google Scholar
  14. Gjestland, L. (1996). Net? Not yet. Mark. Res.: Mag. Manage. Appl. 8: 26–29.Google Scholar
  15. Goree, C. T., and Marszalek, J. F. (1995). Electronic surveys: Ethical issues for researchers. J. Coll. Stud. Affair. 15: 75–79.Google Scholar
  16. Grandy, J. (1998). Response bias in a survey of Asian-American and white science and engineering students. J. Women Minor. Sci. Eng. 4: 1–13.Google Scholar
  17. Hamilton, J. C. (December 30, 1999). The ethics of conducting social-science research on the Internet. The Chronicle of Higher Education, p. B6.Google Scholar
  18. Handwerk, P. G., Carson, C., and Blackwell, K. M. (2000). Online vs. paper-and-pencil surveying of students: A case study. Paper presented at the 40th Annual Forum of the Association for Institutional Research, Cincinnati, OH, May 21–24, 2000.Google Scholar
  19. Hutchison, J., Tollefson, N., and Wigington, H. (1987). Response bias in college freshman's responses to mail surveys. Res. Higher Educ. 26: 99–106.Google Scholar
  20. Johnson, L. C., Beaton, R., Murphy, S., and Pike, K. (2000). Sampling bias and other methodological threats to the validity of health survey research. Int. J. Stress Manage. 7: 247–267.Google Scholar
  21. Kehoe, C. M., and Pitkow, J. E. (1996). Surveying the territory: GVU's five WWW user surveys. W3J 1: 77–84.Google Scholar
  22. Kelley, D. L. (1999). Measurement Made Accessible, Sage Publications, Thousand Oaks.Google Scholar
  23. Krosnick, J. A. (1999). Survey research. Annu. Rev. Psychol. 50: 537–567.Google Scholar
  24. Matz, C. M. (1999). Administration of web versus paper surveys: Mode effects and response rates. Master's Research Paper, University of North Carolina, Chapel Hill.Google Scholar
  25. Newburger, E. C. (2001). Home computers and Internet use in the United States: August 2000. U.S. Census Bureau, Washington, DC. Report retrieved from the World Wide Web at http://www.census.gov/prod/2001pubs/p23-207.pdf.Google Scholar
  26. Palmquist, J., and Stueve, A. (1996). Stay plugged into new opportunities. Mark. Res.: Mag. Manage. Appl. 8: 13–15.Google Scholar
  27. Sax, L. J., Ceja, M., and Teranishi, R. T. (2001). Technological preparedness among entering freshmen: The role of race, class, and gender. J. Educ. Comput. Res. 24: 1–21.Google Scholar
  28. Schmidt, W. C. (1997). World Wide Web survey research: Benefits, potential problems, and solutions. Behav. Res. Methods Instrum. Comput. 29: 274–279.Google Scholar
  29. Schuman, H. (2002). Sense and nonsense about surveys. Contexts 1: 40–47.Google Scholar
  30. Smith, C. B. (1997). Casting the net: Surveying an Internet population. J. Comput.-Mediat. Commun. 3: 77–84.Google Scholar
  31. Smith, M. A., and Leigh, B. (1997). Virtual subjects: Using the Internet as an alternative source of subjects and research environment. Behav. Res. Methods Instrum. Comput. 29: 496–505.Google Scholar
  32. Smith, T. (1995). Trends in nonresponse rates. Int. J. Pub. Opin. Res. 7: 157–171.Google Scholar
  33. Smith, T. W. (1983). The hidden 25 percent: An analysis of nonresponse on the 1980 General Social Survey. In: Singer, E., and Presser, S. (eds.), Survey Research Methods: A Reader, The University of Chicago Press, Chicago, pp. 50–68.Google Scholar
  34. Steeh, C. (1981). Trends in nonresponse rates. Public Opin. Q. 45: 40–57.Google Scholar
  35. Stinchcombe, A. L., Jones, C., and Sheatsley, P. (1981). Nonresponse bias for attitude questions. Public Opin. Q. 45: 359–375.Google Scholar
  36. Tomsic, M. L., Hendel, D. D., and Matross, R. P. (2000). A World Wide Web response to student satisfaction surveys: Comparisons using paper and Internet formats. Paper presented at the 40th Annual Meeting of the Association for Institutional Research, Cincinnati, OH, May 21–24, 2000.Google Scholar
  37. Underwood, D., Kim, H., and Matier, M. (2000). To mail or to Web: Comparisons of survey response rates and respondent characteristics. Paper presented at the 40th Annual Forum of the Association for Institutional Research, Cincinnati, OH, May 21–24, 2000.Google Scholar

Copyright information

© Human Sciences Press, Inc. 2003

Authors and Affiliations

  • Linda J. Sax
    • 1
  • Shannon K. Gilmartin
    • 2
  • Alyssa N. Bryant
    • 2
  1. 1.Higher Education Research InstituteUniversity of California, Los AngelesLos Angeles
  2. 2.Higher Education Research Institute, Graduate School of Education & Information StudiesUniversity of CaliforniaLos Angeles

Personalised recommendations