Using the European Health Interview Survey (EHIS) questionnaire, a web-based survey was organized alongside a face-to-face (F2F) survey including a paper-and-pencil (P&P) questionnaire for sensitive topics. Associated with these different modes, other design features varied too (e.g., recruitment, incentives, sampling). We assessed whether these whole data collection systems developed around the modes produced equivalent health estimates.
Data were obtained from two population-based surveys: the EHISWEB (web-administered, n = 1010) and the Belgian Health Interview Survey 2018 (BHIS2018) (interviewer-administered, n = 2748). Logistic regression analyses were used to assess mode system differences while adjusting for socio-demographic differences in the net samples.
For the P&P mode of the BHIS, significant mode system differences were detected for 2 of the 9 health indicators. Among the indicators collected via the F2F mode, 9 of the 18 indicators showed significant differences.
Indicators collected via the web-based and P&P self-administered modes were generally more comparable than indicators collected via the web-based and F2F mode. Furthermore, fewer differences were detected for indicators based on simple and factual questions compared to indicators based on subjective or complex questions.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Adams SA, Matthews CE, Ebbeling CB et al (2005) The effect of social desirability and social approval on self-reports of physical activity. Am J Epidemiol 161:389–398. https://doi.org/10.1093/aje/kwi054
Biemer PP, Lyberg LE (2003) Introduction to survey quality. Wiley, Hoboken
Bowling A (2005) Mode of questionnaire administration can have serious effects on data quality. J Public Health 27:281–291. https://doi.org/10.1093/pubmed/fdi031
Braekman E, Berete F, Charafeddine R et al (2018) Measurement agreement of the self-administered questionnaire of the Belgian Health Interview Survey: paper-and-pencil versus web-based mode. PLoS ONE 13:e0197434. https://doi.org/10.1371/journal.pone.0197434
Braekman E, Charafeddine R, Demarest S et al (2019) Is the European Health Interview Survey online yet? Response and net sample composition of a web-based data collection. Eur J Public Health. https://doi.org/10.1093/eurpub/ckz206
Burkill S, Copas A, Couper MP et al (2016) Using the web to collect data on sensitive behaviours: a study looking at mode effects on the British National Survey of Sexual Attitudes and Lifestyles. PLoS ONE 11:e0147983. https://doi.org/10.1371/journal.pone.0147983
Christensen AI, Ekholm O, Glümer C et al (2014) Effect of survey mode on response patterns: comparison of face-to-face and self-administered modes in health surveys. Eur J Public Health 24:327–332. https://doi.org/10.1093/eurpub/ckt067
Cox B, Van Oyen H, Cambois E et al (2009) The reliability of the minimum European health module. Int J Public Health 54:55–60. https://doi.org/10.1007/s00038-009-7104-y
De Leeuw ED (2008) Choosing the method of data collection. In: De Leeuw ED, Hox JJ, Dillman DA (eds) International handbook of survey methodology. Lawrence Erlbaum Associates, New York, pp 113–135
De Leeuw ED, Hox JJ, Dillman DA (2008) Mixed-mode surveys: when and why. In: De Leeuw ED, Hox JJ, Dillman DA (eds) International handbook of survey methodology. Lawrence Erlbaum Associates, New York, pp 299–316
Demarest S, Berete F, Charafeddine R, Van der Heyden J. Gezondheidsenquête 2018: Methodologie. Report number: D/2019/14.440/34. Sciensano, Brussels. https://www.gezondheidsenquete.be. Accessed 6 November 2019.
Demarest S, Van der Heyden J, Charafeddine R et al (2013) Methodological basics and evolution of the Belgian Health Interview Survey 1997–2008. Arch Public Health 71:24. https://doi.org/10.1186/0778-7367-71-24
European Union (2018) Commission Regulation (EU) 2018/255: Implementing Regulation (EC) No 1338/2008 of the European Parliament and of the Council as regards statistics based on the European Health Interview Survey (EHIS). Off J Eur Union 48/12:dd21-2-2018.
Eurostat (2018) European Health Interview Survey (EHIS wave 3) methodological manual. Publication Office of the European Union. https://ec.europa.eu/eurostat/documents/3859598/8762193/KS-02-18-240-EN-N.pdf/5fa53ed4-4367-41c4-b3f5-260ced9ff2f6. Accessed 30 May 2018
Feveile H, Olsen O, Hogh A (2007) A randomized trial of mailed questionnaires versus telephone interviews: response patterns in a survey. BMC Med Res Methodol 7:27. https://doi.org/10.1186/1471-2288-7-27
Finger JD, Tafforeau J, Gisle L et al (2015) Development of the European Health Interview Survey-Physical Activity Questionnaire (EHIS-PAQ) to monitor physical activity in the European Union. Arch Public Health 73:59. https://doi.org/10.1186/s13690-015-0110-z
Gnambs T, Kaspar K (2015) Disclosure of sensitive behaviors across self-administered survey modes: a meta-analysis. Behav Res Methods 47:1237–1259. https://doi.org/10.3758/s13428-014-0533-4
Hoebel J, von der Lippe E, Lange C et al (2014) Mode differences in a mixed-mode health interview survey among adults. Arch Public Health 72:46. https://doi.org/10.1186/2049-3258-72-46
Holbrook A, Cho YI, Johnson T (2006) The impact of question and respondent characteristics on comprehension and mapping difficulties. Public Opin Q 70:565–595. https://doi.org/10.1093/poq/nfl027
Keeter S, McGeeney K, Igielnik R et al (2015) From telephone to the web: the challenge of mode of interview effects in public opinion polls. Pew Research Center. https://www.pewresearch.org/wp-content/uploads/2018/08/2015-05-13_mode-study_REPORT.pdf. Accessed 30 May 2019
Kralj M, Zaletel M, Lavtar D et al (2015) Do respondents answer differently in web survey than in face-to-face interview: field work experiment from the European Health Interview Survey (EHIS). https://www.ine.es/q2016/docs/q2016Final00137.pdf. Accessed 30 May 2019
Kroenke K, Spitzer RL, Williams JB (2001) The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 16:606–613. https://doi.org/10.1046/j.1525-1497.2001.016009606.x
Krosnick JA (1991) Response strategies for coping with the cognitive demands of attitude measures in surveys. Appl Cogn Psychol 5:213–236. https://doi.org/10.1002/acp.2350050305
Meltzer H (2003) Development of a common instrument for mental health. In: Nosikov A, Gudex C (eds) EUROHIS: developing common instruments for health surveys. IOS Press, Amsterdam, pp 35–60
Oehlert GW (1992) A note on the Delta method. Am Stat 46:27–29. https://doi.org/10.1080/00031305.1992.10475842
Santourian A, Kitromilidou S (2018) Quality report of the second wave of the European Health Interview Survey. Publication Office of the European Union. https://ec.europa.eu/eurostat/documents/7870049/8920155/KS-FT-18-003-EN-N.pdf/eb85522d-bd6d-460d-b830-4b2b49ac9b03. Accessed 30 May 2019
Schnell R, Noack M, Torregroza S (2017) Differences in general health of internet users and non-users and implications for the use of web surveys. Surv Res Methods 11:105–123. https://doi.org/10.18148/srm/2017.v11i2.6803
Schwarz N, Hippler H-J (1991) Response alternatives: the impact of their choice and presentation order. In: Biemer PP, Groves RM, Lyberg LE, Mathiowetz NA, Sudman S (eds) Measurement errors in surveys. Wiley, New York, pp 41–56
Struminskaya B, de Leeuw ED, Kaczmirek L (2015) Mode system effects in an online panel study: comparing a probability-based online panel with two face-to-face reference surveys. Methods Data Anal 9:3–56. https://doi.org/10.12758/mda.2015.001
Tagseth M, Sund ER, Hallman GT et al (2019) May telephone surveys provide reliable public health surveillance data for municipalities? Mode effects differ between categories of questions. The HUNT Study. Norway. Norsk Epidemiologi 28:105–116. https://doi.org/10.5324/nje.v28i1-2.3057
Tipping S, Hope S, Pickering K et al (2010) The effect of mode and context on survey results: analysis of data from the Health Survey for England 2006 and the Boost Survey for London. BMC Med Res Methodol 10:84. https://doi.org/10.1186/1471-2288-10-84
Tourangeau R, Conrad FG, Couper MP (2013) The science of web surveys. Oxford University Press, New York
Tourangeau R, Rips LJ, Rasinski K (2000) The psychology of survey response. Cambridge University Press, Cambridge
Vergnaud AC, Touvier M, Méjean C et al (2011) Agreement between web-based and paper versions of a socio-demographic questionnaire in the NutriNet-Santé study. Int J Public Health 56:407–417. https://doi.org/10.1007/s00038-011-0257-5
Verschuuren M, Gissler M, Kilpeläinen K et al (2013) Public health indicators for the EU: the joint action for ECHIM (European Community Health Indicators & Monitoring). Arch Public Health 71:12. https://doi.org/10.1186/0778-7367-71-12
Ware JE, Sherbourne CD (1992) The MOS 36-item short-form health survey (SF-36): I. Conceptual framework and item selection. Med care 30:473–483
Washington group (2012) Washington group on disability statistics (WG), Budapest Initiative (BI) & United Nations Economic & Social Commission for Asia & the Pacific (UNESCAP). Development of disability measures for surveys: the extended set on functioning. https://www.washingtongroup-disability.com/wp-content/uploads/2016/01/Development_of_Disability_Measures_for_Surveys_The_Extended_Set_on_Functioning.pdf. Accessed 30 May 2019
Weisband S, Kiesler S (1996) Self disclosure on computer forms: meta-analysis and implications. In: Bilger R, Guest S, Tauber MJ (eds) CHI‘96 Electronic Proceedings. Association for Computing Machinery Inc, New York, pp 1–14
We wish to thank the EHISWEB and BHIS2018 study participants.
This research project was funded by the Belgian Science Policy (Belspo).
Conflict of interest
The authors declare that they have no conflict of interest.
All procedures performed in these studies were in accordance with the ethical standards of the Ethics Committee of the University Hospital of Ghent.
Informed consent was obtained from all individual participants included in the study.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Braekman, E., Charafeddine, R., Demarest, S. et al. Comparing web-based versus face-to-face and paper-and-pencil questionnaire data collected through two Belgian health surveys. Int J Public Health 65, 5–16 (2020). https://doi.org/10.1007/s00038-019-01327-9
- Health surveys
- Data collection
- Web-based surveys
- Face-to-face surveys
- Mode systems
- Data comparability