Abstract
Objectives
To systematically review the literature and compare response rates (RRs) of web surveys to alternative data collection methods in the context of epidemiologic and public health studies.
Methods
We reviewed the literature using PubMed, LILACS, SciELO, WebSM, and Google Scholar databases. We selected epidemiologic and public health studies that considered the general population and used two parallel data collection methods, being one web-based. RR differences were analyzed using two-sample test of proportions, and pooled using random effects. We investigated agreement using Bland-and-Altman, and correlation using Pearson’s coefficient.
Results
We selected 19 studies (nine randomized trials). The RR of the web-based data collection was 12.9 percentage points (p.p.) lower (95% CI = − 19.0, − 6.8) than the alternative methods, and 15.7 p.p. lower (95% CI = − 24.2, − 7.3) considering only randomized trials. Monetary incentives did not reduce the RR differences. A strong positive correlation (r = 0.83) between the RRs was observed.
Conclusions
Web-based data collection present lower RRs compared to alternative methods. However, it is not recommended to interpret this as a meta-analytical evidence due to the high heterogeneity of the studies.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Akmatov MK, Rübsamen N, Schultze A et al (2015) Diverse recruitment strategies result in different participation percentages in a web-based study, but in similar compliance. Int J Public Health 60:937–943. https://doi.org/10.1007/s00038-015-0737-0
Andersson G, Lindvall N, Hursti T, Carlbring P (2002) Hypersensitivity to sound (hyperacusis): a prevalence study conducted via the Internet and post. Int J Audiol 41:545–554
Bälter KA, Bälter O, Fondell E, Lagerros YT (2005) Web-based and mailed questionnaires: a comparison of response rates and compliance. Epidemiol Camb Mass 16:577–579
Barrios M, Villarroya A, Borrego A, Olle C (2011) Response rates and data quality in web and mail surveys administered to PhD holders. Soc Sci Comput Rev 29:208–220. https://doi.org/10.1177/0894439310368031
Beebe TJ, McAlpine DD, Ziegenfuss JY et al (2012) Deployment of a mixed-mode data collection strategy does not reduce nonresponse bias in a general population health survey. Health Serv Res 47:1739–1754. https://doi.org/10.1111/j.1475-6773.2011.01369.x
Bethell C, Fiorillo J, Lansky D et al (2004) Online consumer surveys as a methodology for assessing the quality of the united states health care system. J Med Internet Res 6:e2. https://doi.org/10.2196/jmir.6.1.e2
Bethlehem JG, Biffignandi S (2012) Handbook of web surveys. Wiley, Hoboken
Biemer PP (2010) Total survey error: design, implementation, and evaluation. Public Opin Q 74:817–848. https://doi.org/10.1093/poq/nfq058
Bland JM, Altman DG (1999) Measuring agreement in method comparison studies. Stat Methods Med Res 8:135–160
Blumberg SJ, Ganesh N, Luke JV, Gonzales G (2013) Wireless substitution: state-level estimates from the National Health Interview Survey, 2012. Natl Health Stat Rep 70:1–16
Braunsberger K, Wybenga H, Gates R (2007) A comparison of reliability between telephone and web-based surveys. J Bus Res 60:758–764. https://doi.org/10.1016/j.jbusres.2007.02.015
Brice A, Price A, Burls A (2015) Creating a database of Internet-based clinical trials to support a public-led research programme: a descriptive analysis. Digit Health. https://doi.org/10.1177/2055207615617854
Brown O, Quick V, Colby S et al (2015) Recruitment lessons learned from a tailored web-based health intervention Project Y.E.A.H. (Young Adults Eating and Active for Health). Health Educ 115:470–479. https://doi.org/10.1108/he-06-2014-0075
Dillman DA, Smyth JD, Christian LM (2014) Internet, phone, mail, and mixed-mode surveys: the tailored design method, 4th edn. Wiley, Hoboken
Dinitto DM, Bridget Busch-Armendariz N, Bender K et al (2008) Testing telephone and web surveys for studying men’s sexual assault perpetration behaviors. J Interpers Violence 23:1483–1493. https://doi.org/10.1177/0886260508314341
Edwards PJ, Roberts I, Clarke MJ et al (2009) Methods to increase response to postal and electronic questionnaires. In: The Cochrane Collaboration (ed) Cochrane database of systematic reviews. Wiley, Chichester
Gajic A, Cameron D, Hurley J (2012) The cost-effectiveness of cash versus lottery incentives for a web-based, stated-preference community survey. Eur J Health Econ HEPAC Health Econ Prev Care 13:789–799. https://doi.org/10.1007/s10198-011-0332-0
Galea S, Tracy M (2007) Participation rates in epidemiologic studies. Ann Epidemiol 17:643–653. https://doi.org/10.1016/j.annepidem.2007.03.013
Greenlaw C, Brown-Welty S (2009) A comparison of web-based and paper-based survey methods: testing assumptions of survey mode and response cost. Eval Rev 33:464–480. https://doi.org/10.1177/0193841x09340214
Groves RM (2004) Survey errors and survey costs. Wiley, Hoboken
Groves RM, Peytcheva E (2008) The impact of nonresponse rates on nonresponse bias: a meta-analysis. Public Opin Q 72:167–189. https://doi.org/10.1093/poq/nfn011
Haider S, Dodge LE, Brown BA et al (2013) Evaluation of e-mail contact to conduct follow-up among adolescent women participating in a longitudinal cohort study of contraceptive use. Contraception 88:18–23. https://doi.org/10.1016/j.contraception.2012.11.016
Hardigan PC, Succar CT, Fleisher JM (2012) An analysis of response rate and economic costs between mail and web-based surveys among practicing dentists: a randomized trial. J Community Health 37:383–394. https://doi.org/10.1007/s10900-011-9455-6
Higgins JPT, Thompson SG (2002) Quantifying heterogeneity in a meta-analysis. Stat Med 21:1539–1558. https://doi.org/10.1002/sim.1186
Hohwü L, Lyshol H, Gissler M et al (2013) Web-based versus traditional paper questionnaires: a mixed-mode survey with a nordic perspective. J Med Internet Res 15:e173. https://doi.org/10.2196/jmir.2595
Holland CM, Ritchie ND, Du Bois SN (2015) iTunes song-gifting is a low-cost, efficient recruitment tool to engage high-risk MSM in Internet research. AIDS Behav 19:1914–1918. https://doi.org/10.1007/s10461-015-1130-y
Huybrechts KF, Mikkelsen EM, Christensen T et al (2010) A successful implementation of e-epidemiology: the Danish pregnancy planning study “Snart-Gravid”. Eur J Epidemiol 25:297–304. https://doi.org/10.1007/s10654-010-9431-y
Källmén H, Sinadinovic K, Berman A, Wennberg P (2011) Risky drinking of alcohol in Sweden: a randomized population survey comparing web- and paper-based self-reports. Nord Stud Alcohol Drugs. https://doi.org/10.2478/v10199-011-0013-4
Keiding N, Louis TA (2016) Perils and potentials of self-selected entry to epidemiological studies and surveys. J R Stat Soc Ser A (Statistics in Society) 179:319–376. https://doi.org/10.1111/rssa.12136
Klausch T, Hox JJ, Schouten B (2013) Measurement effects of survey mode on the equivalence of attitudinal rating scale questions. Sociol Methods Res 42:227–263. https://doi.org/10.1177/0049124113500480
Laaksonen S, Heiskanen M (2014) Comparison of three modes for a crime victimization survey. J Surv Stat Methodol 2:459–483. https://doi.org/10.1093/jssam/smu018
Lagerros YT, Sandin S, Bexelius C et al (2012) Estimating physical activity using a cell phone questionnaire sent by means of short message service (SMS): a randomized population-based study. Eur J Epidemiol 27:561–566. https://doi.org/10.1007/s10654-012-9708-4
Lee C-K, Back K-J, Williams RJ, Ahn S-S (2015) Comparison of telephone RDD and online panel survey modes on CPGI scores and co-morbidities. Int Gambl Stud 15:435–449. https://doi.org/10.1080/14459795.2015.1068353
Link MW, Mokdad AH (2005) Alternative modes for health surveillance surveys: an experiment with web, mail, and telephone. Epidemiol Camb Mass 16:701–704
Lozar Manfreda K, Bosnjak M, Berzelak J et al (2008) Web surveys versus other survey modes: a meta-analysis comparing response rates. Int J Mark Res 50:79–104
McCormack LA, Friedrich C, Fahrenwald N, Specker B (2014) Feasibility and acceptability of alternate methods of postnatal data collection. Matern Child Health J 18:852–857. https://doi.org/10.1007/s10995-013-1310-1
McMillen RC, Winickoff JP, Wilson K et al (2013) A dual-frame sampling methodology to address landline replacement in tobacco control research. Tob Control 24:7–10. https://doi.org/10.1136/tobaccocontrol-2012-050727
Morton LM, Cahill J, Hartge P (2006) Reporting participation in epidemiologic studies: a survey of practice. Am J Epidemiol 163:197–203. https://doi.org/10.1093/aje/kwj036
Pew Research Center (2016) Smartphone ownership and internet usage continues to climb in emerging economies. Pew Research Center
Sax LJ, Gilmartin SK, Bryant AN (2003) Assessing response rates and nonresponse bias in web and paper surveys. Res High Educ 44:409–432. https://doi.org/10.1023/a:1024232915870
Schonlau M, Zapert K, Simon LP et al (2004) A comparison between responses from a propensity-weighted web survey and an identical RDD survey. Soc Sci Comput Rev 22:128–138. https://doi.org/10.1177/0894439303256551
Sebo P, Maisonneuve H, Cerutti B et al (2017) Rates, delays, and completeness of general practitioners’ responses to a postal versus web-based survey: a randomized trial. J Med Internet Res 19:e83. https://doi.org/10.2196/jmir.6308
Shih T-H, Fan Xitao (2008) Comparing response rates from web and mail surveys: a meta-analysis. Field Methods 20:249–271. https://doi.org/10.1177/1525822x08317085
Shin E, Johnson TP, Rao K (2012) Survey mode effects on data quality: comparison of web and mail modes in a U.S. National Panel Survey. Soc Sci Comput Rev 30:212–228. https://doi.org/10.1177/0894439311404508
Sinadinovic K, Wennberg P, Berman AH (2011) Population screening of risky alcohol and drug use via Internet and interactive voice response (IVR): a feasibility and psychometric study in a random sample. Drug Alcohol Depend 114:55–60. https://doi.org/10.1016/j.drugalcdep.2010.09.004
Spijkerman R, Knibbe R, Knoops K et al (2009) The utility of online panel surveys versus computer-assisted interviews in obtaining substance-use prevalence estimates in the Netherlands. Addiction 104:1641–1645. https://doi.org/10.1111/j.1360-0443.2009.02642.x
The American Association for Public Opinion Research (2016) Standard definitions: final dispositions of case codes and outcome rates for surveys, 9th edn. AAPOR
The World Bank (2018) Internet users (per 100 people). World Bank. http://data.worldbank.org/indicator/IT.NET.USER.P2?end=2014&start=2014&view=bar. Accessed 21 Feb 2018
Tonsaker T, Bartlett G, Trpkov C (2014) Health information on the internet: gold mine or minefield? Can Fam Physician Med Fam Can 60:407–408
United Problem Solvers (UPS) (2016) UPS pulse of the online shopper: tech-savvy shoppers transforming retail—a UPS white paper. United Problem Solvers (UPS)
Van Mol C (2016) Improving web survey efficiency: the impact of an extra reminder and reminder content on web survey response. Int J Soc Res Methodol. https://doi.org/10.1080/13645579.2016.1185255
Wallander L, Tikkanen RH, Mannheimer LN et al (2015) The problem of non-response in population surveys on the topic of HIV and sexuality: a comparative study. Eur J Public Health 25:172–177. https://doi.org/10.1093/eurpub/cku154
Yuen YY, Yeow PHP, Lim N, Saylani N (2010) Internet banking adoption: comparing developed and developing countries. J Comput Inf Syst 51:52–61. https://doi.org/10.1080/08874417.2010.11645449
Funding
No funding was received to execute this study.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflicts of interest.
Ethical statement
This article does not contain any studies with human participants or animals performed by any of the authors.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Blumenberg, C., Barros, A.J.D. Response rate differences between web and alternative data collection methods for public health research: a systematic review of the literature. Int J Public Health 63, 765–773 (2018). https://doi.org/10.1007/s00038-018-1108-4
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00038-018-1108-4