Comparison of response rates and cost-effectiveness for a community-based survey: postal, internet and telephone modes with generic or personalised recruitment approaches
Epidemiological research often requires collection of data from a representative sample of the community or recruitment of specific groups through broad community approaches. The population coverage of traditional survey methods such as mail-outs to residential addresses, and telephone contact via public directories or random-digit-dialing is declining and survey response rates are falling. There is a need to explore new sampling frames and consider multiple response modes including those offered by changes in telecommunications and internet technology.
We evaluated response rates and cost-effectiveness for three modes of survey administration (postal invitation/postal survey, postal invitation/internet survey and postal invitation/telephone survey) and two styles of contact approach (personalised and generic) in a community survey of greywater use. Potential respondents were contacted only once, with no follow up of non-responders.
The telephone survey produced the highest adjusted response rate (30.2%), followed by the personalised postal survey (10.5%), generic postal survey (7.5%) and then the internet survey (4.7% for the personalised approach and 2.2% for the generic approach). There were some differences in household characteristics and greywater use rates between respondents to different survey modes, and between respondents to personalised and generic approaches. These may be attributable to the differing levels of motivations needed for a response, and varying levels of interest in the survey topic among greywater users and non-users. The generic postal survey had the lowest costs per valid survey received (Australian $22.93), followed by the personalised postal survey ($24.75).
Our findings suggest that postal surveys currently remain the most economic option for population-based studies, with similar costs for personalised and generic approaches. Internet surveys may be effective for specialised groups where email lists are available for initial contact, but barriers other than household internet access still exist for community-based surveys. Given the increasing recruitment challenges facing community-based studies, there is an imperative to gather contemporary comparative data on different survey modes and recruitment approaches in order to determine their strengths, limitations and costs. Researchers also need to document and report on the potential biases in the target and respondent populations and how this may affect the data collected.
- Blumberg SJ, Luke JV: Wireless substitution: early release of estimates from the National Health Interview Survey, July–December 2009. Centers for Disease Control and Prevention; 2010. http://www.cdc.gov/nchs/nhis/releases.htm#wireless
- Blumberg SJ, Luke JV: Wireless substitution: Early release of estimates from the National Health Interview Survey, January-June 2011. Centers for Disease Control and Prevention; 2011. http://www.cdc.gov/nchs/nhis/releases.htm#wireless
- Guterbock TM, Diop A, Ellis JM, Holmes JL, Le KT: Who needs RDD? Combining directory listings with cell phone exchanges for an alternative telephone sampling frame. Soc Sci Res 2011,40(3):860–872. CrossRef
- Galea S, Tracy M: Participation rates in epidemiologic studies. Ann Epidemiol 2007, 17:643–653. CrossRef
- Curtin R, Presser S, Singer E: Changes in Telephone Survey Nonresponse over the Past Quarter Century. Public Opin Q 2005,69(1):87–98. CrossRef
- Ekman A, Litton J-E: New times, new needs; e-epidemiology. Eur J Epidemiol 2007, 22:285–292. CrossRef
- Van Gelder MMHJ, Bretveld RW, Roeleveld N: Web-based questionnaires: the future in epidemiology? Am J Epidemiol 2010,172(11):1292–1298. CrossRef
- Link MW, Mokdad AH: Alternative modes for health surveillance surveys: an experiment with web, mail and telephone. Epidemiology 2005,16(5):701–704. CrossRef
- Källmén H, Sinadinovic K, Berman AH, Wennberg P: Risky drinking of alcohol in Sweden: a randomized population survey comparing web- and paper-based self-reports. NAT Nordic Stud Alcohol and Drugs 2011,28(2):123–130. CrossRef
- Eriksen L, Grønbæk M, Helge JW, Tolstrup JS, Curtis T: The Danish health examination survey 2007–2008 (DANHES 2007–2008). Scand J Public Health 2011,39(2):203–211. CrossRef
- Manfreda KL, Bosnjak M, Berzelak J, Haas I, Vehovar V: Web surveys versus other survey modes. A meta-analysis comparing response rates. Int J Market Res 2008,50(1):79–104.
- Shih TH, Xitao F: Comparing response rates from web and mail surveys: A meta-analysis. Field Methods 2008,20(3):249–271. CrossRef
- Balter KA, Balter O, Fondell E, Lagerros YT: Web-based and mailed questionnaires: A comparison of response rates and compliance. Epidemiology 2005,16(4):577–579. CrossRef
- Ekman A, Dickman PW, sa Klint A, Weiderpass E, Litton JE: Feasibility of using web-based questionnaires in large population-based epidemiological studies. Eur J Epidemiol 2006, 21:103–111. CrossRef
- Couper MP: The future of modes of data collection. Public Opin Q 2011,75(5 SPEC):889–908. CrossRef
- Blyth B: Mixed mode: The only ‘fitness’ regime? Int J Market Res 2008,50(2):241–266.
- Brogger J, Nystad W, Cappelen I, Bakke P: No increase in response rate by adding a web response option to a postal population survey: a randomized trial. J Med Internet Res 2007,9(5):e40. CrossRef
- Turunen M, Paanala A, Villman J, Nevalainen A, Haverinen-Shaughnessy U: Evaluating housing quality, health and safety using an Internet-based data collection and response system: a cross-sectional study. Environ Health 2010, 9:69. CrossRef
- Messer BL, Dillman DA: Surveying the general public over the internet using address-based sampling and mail contact procedures. Public Opin Q 2011,75(3):429–457. CrossRef
- Dillman DA, Phelps G, Tortora R, Swift K, Kohrell J, Berck J, Messer BL: Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet. Soc Sci Res 2009,38(1):1–18. CrossRef
- Fan W, Yan Z: Factors affecting response rates of the web survey: a systematic review. Comput Hum Behav 2010, 26:132–139. CrossRef
- Edwards PJ, Roberts I, Clarke MJ, DiGuiseppi C, Wentz R, Kwan I, Cooper R, Felix LM, Pratap S: Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev 2009, Issue 3:Art. No.:MR000008. doi:10.1002/14651858.MR000008.pub4.
- Dillman DA, Lesser V, Mason R, Carlson J, Willits F, Robertson R, Burke B: Personalization of mail surveys for general public and populations with a group identity: results from nine studies. Rural Sociol 2007,72(4):632–646. CrossRef
- Heerwegh D: Effects of personal salutations in e-mail invitations to participate in a web survey. Public Opin Q 2005,69(4):588–598. CrossRef
- Muñoz-Leiva F, Sánchez-Fernández J, Montoro-Ríos F, Ibáñez-Zapata JA: Improving the response rate and quality in Web-based surveys through the personalization and frequency of reminder mailings. Qual Quant 2010,44(5):1037–1052. CrossRef
- Sauermann H, Roach M: Increasing web survey response rates in innovation research: An experimental study of static and dynamic contact design features. Res Policy 2012. http://dx.doi.org/10.1016/j.respol.2012.05.003
- Gajic A, Cameron D, Hurley J: The cost-effectiveness of cash versus lottery incentives for a web-based, stated-preference community survey. Eur J Health Econ 2011, 1–11. doi:10.1007/s10198-011-0332-0.
- Göritz AS: Incentives in web studies: methodological issues and a review. Int J Internet Sci 2006,1(1):58–70.
- ABS: Socio-Economic Indexes for Areas. Australian Bureau of Statistics; 2008. http://www.abs.gov.au/websitedbs/D3310114.nsf/home/Seifa_entry_page
- ABS: Census Home Page. Australian Bureau of Statistics; 2006. http://www.abs.gov.au/websitedbs/censushome.nsf/home/Census
- De Leeuw E, Callegaro M, Hox J, Korendijk E, Lensvelt-Mulders G: The influence of advance letters on response in telephone surveys. A meta-analysis. Public Opin Q 2007,71(3):413–443. CrossRef
- McCluskey S, Topping AE: Increasing response rates to lifestyle surveys: a pragmatic evidence review. Perspect Public Health 2011,131(2):89–94. CrossRef
- ABS: Environmental Issues: Water use and conservation. Australian Bureau of Statistics; 2010. http://www.abs.gov.au/ausstats/abs@.nsf/mf/4602.0.55.003/
- Mokdad AH: The behavioral risk factors surveillance system: Past, present, and future. Annu Rev Public Health 2009, 30:43–54. CrossRef
- Fenner Y, Garland SM, Moore EE, Jayasinghe Y, Fletcher A, Tabrizi SN, Gunasekaran B, Wark JD: Web-based recruiting for health research using a social networking site: an exploratory study. J Med Internet Res 2012,14(1):e20. doi:10.2196/jmir.1978. CrossRef
- Ramo DE, Hall SM, Prochaska JJ: Reaching young adult smokers through the Internet: Comparison of three recruitment mechanisms. Nicotine Tob Res 2010,12(7):768–775. CrossRef
- Groves RM: Nonresponse rates and nonresponse bias in household surveys. Public Opin Q 2006,70(5):646–675. CrossRef
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2288/12/132/prepub
- Comparison of response rates and cost-effectiveness for a community-based survey: postal, internet and telephone modes with generic or personalised recruitment approaches
- Open Access
- Available under Open Access This content is freely available online to anyone, anywhere at any time.
BMC Medical Research Methodology
- Online Date
- August 2012
- Online ISSN
- BioMed Central
- Additional Links
- Survey methods
- Postal survey
- Telephone survey
- Internet survey
- Author Affiliations
- 1. Department of Epidemiology and Preventive Medicine, School of Public Health and Preventive Medicine, Monash University, Melbourne, Australia
- 2. School of Land and Environment, The University of Melbourne, Parkville, Australia