Skip to main content

A comparison of nonresponse in mail, telephone, and face-to-face surveys

Applying multilevel modeling to meta-analysis

Abstract

This article reports a meta-analysis of 45 studies that explicitly compare the response obtained using a mail, telephone or face-to-face survey. The data analysis uses a generalized hierarchical linear model. Sampling procedure (e.g., local convenience sample, random general sample), saliency of topic, and research organization (university, government versus market research) had an effect on the response. On the average, the face-to-face condition achieved the highest completion rate (70.3%), the telephone survey the next highest (67.2%), and the mail survey the lowest (61.3%). There is a significant interaction with the year of publication: The response to face-to-face and telephone surveys is going down in the period covered by this analysis (1947 to 1992), but the response to mail surveys is going up slightly. We attribute this to the large amount of research on nonresponse problems with mail surveys, and recommend more research and development in this direction for face-to-face and telephone methods.

This is a preview of subscription content, access via your institution.

References

  1. Bangert-Drowns, R. L. (1986). Review of developments in meta-analytic method,Psychological Bulletin 99: 388–399.

    Google Scholar 

  2. Betlehem, J. G. & Kersten, H. M. P. (1981). The nonresponse problem,Survey Methodology 7: 130–156.

    Google Scholar 

  3. Bryk, A. S. & Raudenbusch, S. W. (1992).Hierarchical Linear Models: Applications and Data Analysis Methods. Newbury Park: Sage.

    Google Scholar 

  4. De Leeuw, E. D. (1992).Data Quality in Mail, Telephone and Face-to-Face Surveys. Amsterdam: Vrije Universiteit (Doctoral dissertation).

    Google Scholar 

  5. Dillman, D. A. (1978).Mail and Telephone Surveys: The Total Design Method. New York: Wiley.

    Google Scholar 

  6. Dillman, D. A. (1991). The design and administration of mail surveys,Annual Review of Sociology 17: 225–249.

    Google Scholar 

  7. Fox, R. J., Crask, M. R., & Kim, J. (1988). Mail survey response rate; A meta-analysis of selected techniques for inducing response,Public Opinion 52: 467–491.

    Google Scholar 

  8. Goldstein, H. (1987).Multilevel Methods in Educational and Social Research. New York: Oxford University Press.

    Google Scholar 

  9. Goldstein, H. (1991). Nonlinear multilevel models, with an application to discrete response data,Biometrika 78: 45–51.

    Google Scholar 

  10. Glass, G. V., McGaw, B. & Smith, M. L. (1981).Meta-Analysis in Social Research. Beverly Hills: Sage.

    Google Scholar 

  11. Goyder, J. (1987).The Silent Minority. Cambridge: Blackwell.

    Google Scholar 

  12. Groves, R. M. & Kahn, R. L. (1979).Surveys by Telephone, A National Comparison with Personal Interviews. New York: Academic Press.

    Google Scholar 

  13. Groves, R. M. (1989).Survey Errors and Survey Costs. New York: Wiley.

    Google Scholar 

  14. Groves, R. M. & Lyberg, L. E. (1988). An overview of nonresponse issues in telephone surveys, pp. 191–212 in R. M. Groves, P. P. Biemer, L. E. Lyberg, J. T. Massey, W. L. Nicholls II, & J. Waksberg (eds),Telephone Survey Methodology. New York: Wiley.

    Google Scholar 

  15. Heberlein, T. A. & Baumgartner, R. (1978). Factors affecting response rates to mailed questionnaires: A quantitative analysis of the published literature,American Sociological Review 43: 447–462.

    Google Scholar 

  16. Hedges, L. V. & Olkin, I. (1985).Statistical Methods for Meta-Analysis. Orlando: Academic Press.

    Google Scholar 

  17. Hunter, J. E., & Schmidt, F. L. (1990).Methods of Meta-Analysis. Beverly Hills: Sage.

    Google Scholar 

  18. Jaccard, J., Turrisi, R. & Wan, C. K. (1990).Interaction Effects in Multiple Regression. Newbury Park: Sage.

    Google Scholar 

  19. Longford, N. T. (1988).A Quasi-Likelihood Adaptation for Variance Component Analysis. Princeton, NJ: Educational Testing Service.

    Google Scholar 

  20. Longford, N. T. (1990). VARCL.Software for Variance Component Analysis of Data with Nested Random Effects (Maximum Likelihood). Princeton, NJ: Educational Testing Service.

    Google Scholar 

  21. Lyberg, I. & Lyberg, L. (1990). Nonresponse Research at Statistics Sweden. Paper presented at the First Workshop on Household Survey Nonresponse. Stockholm, Oct. 15–17, 1990.

  22. McCullagh, P. & Nelder, J. A. (1989).Generalized Linear Models. London: Chapman & Hall.

    Google Scholar 

  23. Rosenthal, R. (1984).Meta-Analytic Procedures for Social Research. Beverly Hills: Sage.

    Google Scholar 

  24. Rosenthal, R. & Rubin, D. B. (1986). Meta-analytic procedures for combining studies with multiple effect sizes,Psychological Bulletin 99: 400–406.

    Google Scholar 

  25. Steeh, C. G. (1981). Trends in nonresponse rates, 1952–1979,Public Opinion Quarterly, 45. Reprinted as pp. 32–49 in E. Singer & S. Presser (1989),Survey Research Methods, A Reader. Chicago: University of Chicago Press.

    Google Scholar 

  26. Sugiyama, M. (1992). Response and non-response, pp. 227–239 in L. Lebart (Ed.),Quality of Information in Sample Surveys. Paris: Dunod.

    Google Scholar 

  27. Wong, G. Y. & Mason, W. M. (1985). The hierarchical logistic regression model for multilevel analysis. Extensions of the hierarchical normal linear model for multilevel analysis,Journal of the American Statistical Association 80: 513–524.

    Google Scholar 

  28. Yu, J., & Cooper, H. (1983). A quantitative review of research design effects on response rates to questionnaires,Journal of Marketing Research 23: 36–44.

    Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Joop J. Hox.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Hox, J.J., De Leeuw, E.D. A comparison of nonresponse in mail, telephone, and face-to-face surveys. Qual Quant 28, 329–344 (1994). https://doi.org/10.1007/BF01097014

Download citation

Keywords

  • Data Analysis
  • Linear Model
  • Significant Interaction
  • Convenience Sample
  • Sampling Procedure