Comparison of response rates and cost-effectiveness for a community-based survey: postal, internet and telephone modes with generic or personalised recruitment approaches
Epidemiological research often requires collection of data from a representative sample of the community or recruitment of specific groups through broad community approaches. The population coverage of traditional survey methods such as mail-outs to residential addresses, and telephone contact via public directories or random-digit-dialing is declining and survey response rates are falling. There is a need to explore new sampling frames and consider multiple response modes including those offered by changes in telecommunications and internet technology.
We evaluated response rates and cost-effectiveness for three modes of survey administration (postal invitation/postal survey, postal invitation/internet survey and postal invitation/telephone survey) and two styles of contact approach (personalised and generic) in a community survey of greywater use. Potential respondents were contacted only once, with no follow up of non-responders.
The telephone survey produced the highest adjusted response rate (30.2%), followed by the personalised postal survey (10.5%), generic postal survey (7.5%) and then the internet survey (4.7% for the personalised approach and 2.2% for the generic approach). There were some differences in household characteristics and greywater use rates between respondents to different survey modes, and between respondents to personalised and generic approaches. These may be attributable to the differing levels of motivations needed for a response, and varying levels of interest in the survey topic among greywater users and non-users. The generic postal survey had the lowest costs per valid survey received (Australian $22.93), followed by the personalised postal survey ($24.75).
Our findings suggest that postal surveys currently remain the most economic option for population-based studies, with similar costs for personalised and generic approaches. Internet surveys may be effective for specialised groups where email lists are available for initial contact, but barriers other than household internet access still exist for community-based surveys. Given the increasing recruitment challenges facing community-based studies, there is an imperative to gather contemporary comparative data on different survey modes and recruitment approaches in order to determine their strengths, limitations and costs. Researchers also need to document and report on the potential biases in the target and respondent populations and how this may affect the data collected.
- Blumberg, SJ, Luke, JV (2010) Wireless substitution: early release of estimates from the National Health Interview Survey, July–December 2009.
- Blumberg, SJ, Luke, JV (2011) Wireless substitution: Early release of estimates from the National Health Interview Survey, January-June 2011.
- Guterbock, TM, Diop, A, Ellis, JM, Holmes, JL, Le, KT (2011) Who needs RDD? Combining directory listings with cell phone exchanges for an alternative telephone sampling frame. Soc Sci Res 40: pp. 860-872 CrossRef
- Galea, S, Tracy, M (2007) Participation rates in epidemiologic studies. Ann Epidemiol 17: pp. 643-653 CrossRef
- Curtin, R, Presser, S, Singer, E (2005) Changes in Telephone Survey Nonresponse over the Past Quarter Century. Public Opin Q 69: pp. 87-98 CrossRef
- Ekman, A, Litton, J-E (2007) New times, new needs; e-epidemiology. Eur J Epidemiol 22: pp. 285-292 CrossRef
- Van Gelder, MMHJ, Bretveld, RW, Roeleveld, N (2010) Web-based questionnaires: the future in epidemiology?. Am J Epidemiol 172: pp. 1292-1298 CrossRef
- Link, MW, Mokdad, AH (2005) Alternative modes for health surveillance surveys: an experiment with web, mail and telephone. Epidemiology 16: pp. 701-704 CrossRef
- Källmén, H, Sinadinovic, K, Berman, AH, Wennberg, P (2011) Risky drinking of alcohol in Sweden: a randomized population survey comparing web- and paper-based self-reports. NAT Nordic Stud Alcohol and Drugs 28: pp. 123-130 CrossRef
- Eriksen, L, Grønbæk, M, Helge, JW, Tolstrup, JS, Curtis, T (2011) The Danish health examination survey 2007–2008 (DANHES 2007–2008). Scand J Public Health 39: pp. 203-211 CrossRef
- Manfreda, KL, Bosnjak, M, Berzelak, J, Haas, I, Vehovar, V (2008) Web surveys versus other survey modes. A meta-analysis comparing response rates. Int J Market Res 50: pp. 79-104
- Shih, TH, Xitao, F (2008) Comparing response rates from web and mail surveys: A meta-analysis. Field Methods 20: pp. 249-271 CrossRef
- Balter, KA, Balter, O, Fondell, E, Lagerros, YT (2005) Web-based and mailed questionnaires: A comparison of response rates and compliance. Epidemiology 16: pp. 577-579 CrossRef
- Ekman, A, Dickman, PW, sa Klint, A, Weiderpass, E, Litton, JE (2006) Feasibility of using web-based questionnaires in large population-based epidemiological studies. Eur J Epidemiol 21: pp. 103-111 CrossRef
- Couper, MP (2011) The future of modes of data collection. Public Opin Q 75: pp. 889-908 CrossRef
- Blyth, B (2008) Mixed mode: The only ‘fitness’ regime?. Int J Market Res 50: pp. 241-266
- Brogger, J, Nystad, W, Cappelen, I, Bakke, P (2007) No increase in response rate by adding a web response option to a postal population survey: a randomized trial. J Med Internet Res 9: pp. e40 CrossRef
- Turunen, M, Paanala, A, Villman, J, Nevalainen, A, Haverinen-Shaughnessy, U (2010) Evaluating housing quality, health and safety using an Internet-based data collection and response system: a cross-sectional study. Environ Health 9: pp. 69 CrossRef
- Messer, BL, Dillman, DA (2011) Surveying the general public over the internet using address-based sampling and mail contact procedures. Public Opin Q 75: pp. 429-457 CrossRef
- Dillman, DA, Phelps, G, Tortora, R, Swift, K, Kohrell, J, Berck, J, Messer, BL (2009) Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet. Soc Sci Res 38: pp. 1-18 CrossRef
- Fan, W, Yan, Z (2010) Factors affecting response rates of the web survey: a systematic review. Comput Hum Behav 26: pp. 132-139 CrossRef
- Edwards, PJ, Roberts, I, Clarke, MJ, DiGuiseppi, C, Wentz, R, Kwan, I, Cooper, R, Felix, LM, Pratap, S (2009) Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev Issue 3: pp. Art. No.:MR000008
- Dillman, DA, Lesser, V, Mason, R, Carlson, J, Willits, F, Robertson, R, Burke, B (2007) Personalization of mail surveys for general public and populations with a group identity: results from nine studies. Rural Sociol 72: pp. 632-646 CrossRef
- Heerwegh, D (2005) Effects of personal salutations in e-mail invitations to participate in a web survey. Public Opin Q 69: pp. 588-598 CrossRef
- Muñoz-Leiva, F, Sánchez-Fernández, J, Montoro-Ríos, F, Ibáñez-Zapata, JA (2010) Improving the response rate and quality in Web-based surveys through the personalization and frequency of reminder mailings. Qual Quant 44: pp. 1037-1052 CrossRef
- Sauermann, H, Roach, M (2012) Increasing web survey response rates in innovation research: An experimental study of static and dynamic contact design features. Res Policy.
- Gajic, A, Cameron, D, Hurley, J (2011) The cost-effectiveness of cash versus lottery incentives for a web-based, stated-preference community survey. Eur J Health Econ. pp. 1-11
- Göritz, AS (2006) Incentives in web studies: methodological issues and a review. Int J Internet Sci 1: pp. 58-70
- Socio-Economic Indexes for Areas.
- Census Home Page.
- De Leeuw, E, Callegaro, M, Hox, J, Korendijk, E, Lensvelt-Mulders, G (2007) The influence of advance letters on response in telephone surveys. A meta-analysis. Public Opin Q 71: pp. 413-443 CrossRef
- McCluskey, S, Topping, AE (2011) Increasing response rates to lifestyle surveys: a pragmatic evidence review. Perspect Public Health 131: pp. 89-94 CrossRef
- Environmental Issues: Water use and conservation.
- Mokdad, AH (2009) The behavioral risk factors surveillance system: Past, present, and future. Annu Rev Public Health 30: pp. 43-54 CrossRef
- Fenner, Y, Garland, SM, Moore, EE, Jayasinghe, Y, Fletcher, A, Tabrizi, SN, Gunasekaran, B, Wark, JD (2012) Web-based recruiting for health research using a social networking site: an exploratory study. J Med Internet Res 14: pp. e20 CrossRef
- Ramo, DE, Hall, SM, Prochaska, JJ (2010) Reaching young adult smokers through the Internet: Comparison of three recruitment mechanisms. Nicotine Tob Res 12: pp. 768-775 CrossRef
- Groves, RM (2006) Nonresponse rates and nonresponse bias in household surveys. Public Opin Q 70: pp. 646-675 CrossRef
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/12/132/prepub
- Comparison of response rates and cost-effectiveness for a community-based survey: postal, internet and telephone modes with generic or personalised recruitment approaches
- Open Access
- Available under Open Access This content is freely available online to anyone, anywhere at any time.
BMC Medical Research Methodology
- Online Date
- August 2012
- Online ISSN
- BioMed Central
- Additional Links
- Survey methods
- Postal survey
- Telephone survey
- Internet survey
- Author Affiliations
- 1. Department of Epidemiology and Preventive Medicine, School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
- 2. School of Land and Environment, The University of Melbourne, Parkville, Australia