Introduction

Rationale

When collecting information from large, geographically dispersed populations, a self-administered questionnaire is usually the only financially viable option [1]. Non-responses to questionnaires reduce effective sample sizes, reducing study power, and may introduce bias in study results [2]. The Cochrane Methodology Review of methods to increase response to self-administered questionnaires has provided a much-used scientific evidence base for effective data collection by questionnaire since the publication of the first version of the review in 2003 which focused on postal questionnaires [3].

A decade ago paper-and-pencil administration of questionnaires in epidemiological studies was twenty times more common than electronic administration [4], but increased Internet access and decreasing volumes of mailed letters suggests that electronic administration has gained favour [5,6,7]. Researchers planning to collect data from participants using a self-administered questionnaire need to know how will the proportion of participants responding to a questionnaire administered electronically compare with one administered by post? We conducted further analyses of the trials included in the recently updated Cochrane Review [8] to answer this question.

Objective

To assess whether response rates to questionnaires administered electronically differ to those of questionnaires administered by post.

Methods

Data sources/measurement

We exported data of randomised controlled trials included in the updated Cochrane Review [8] from RevMan and imported them into Stata for a series of meta-analyses not conducted in the Cochrane review.

Comparisons

We focused on data from trials included in three comparisons in the Cochrane Review that had evaluated hypotheses relevant to our research objective:

  1. 1.

    Postal vs. electronic questionnaire (Cochrane Comparison 81).

  2. 2.

    Electronic questionnaire only vs. choice (postal or electronic) (Cochrane Comparison 84).

  3. 3.

    Choice (electronic or postal) vs. postal questionnaire only (Cochrane Comparison 82).

These comparisons assess: response to questionnaires administered by post compared with questionnaires administered electronically, response to a questionnaire administered electronically compared with response when including a postal response option, and response when including an electronic response option compared with response to a questionnaire administered by post only, respectively.

Data items

Outcome measures

The data obtained from each trial included the numbers of participants randomised to each arm of the trial with the numbers of completed, or partially completed questionnaires returned after all mailings (for trials including a postal questionnaire), and the numbers of participants randomised to each arm of the trial with the numbers of participants submitting the completed, or partially completed online questionnaires after all contacts (electronic questionnaire).

Other variables

Additional data were also extracted on the:

  • Year of publication of the study.

  • Risk of bias in each included study (a judgment - high, low, or unclear); we assessed the overall risk of bias in each study using the Cochrane Collaboration’s tool [9].

Effect measures and synthesis

For each of the three comparisons (2.1.1 above), we pooled the odds ratios for response in each included study in a random effects meta-analysis (to allow for heterogeneity of effect estimates between studies) using the metan command in Stata [10]. This command also produced a forest plot (a visual display of the results of the individual studies and syntheses) for each comparison. We quantified any heterogeneity using the I2 statistic that describes the percentage of the variability in effect estimates that is due to heterogeneity [11].

Subgroup analyses

We explored possible causes of heterogeneity among study results by conducting subgroup analyses according to two study-level factors: Year of study publication, and risk of bias in studies. We used a statistical test of homogeneity of the pooled effects in subgroups to assess evidence for subgroup differences. The statistical test of homogeneity used is Cochran’s Q test, where the Q statistic is distributed as a chi-square statistic with k-1 degrees of freedom, where k is the number of subgroups. If there was evidence for subgroup differences provided by the test of homogeneity, we chose the ‘best estimate of effect’ as the estimate from the subgroup of studies with low risk of bias, or the subgroup of studies published after 2012. If there was no evidence for subgroup differences, we chose our best estimate of effect based on the synthesis of all studies.

Year of study publication

From 2012, household access to a computer exceeded 40%: [5] As the odds ratios for response to questionnaires administered electronically may be associated with household access to a computer, we analysed trial results in two subgroups – before 2012 and after 2012, where we used the year of publication as an approximation of the year of study conduct.

Risk of bias

The odds ratios for response estimated in the included studies may be associated with trial quality. [12, 13] For this reason we analysed trial results in two subgroups – trials judged to be at low and at high risk of bias.

Reporting bias assessment

We assessed evidence for reporting bias using Harbord’s modified test for small-study effects implemented in Stata using the metabias command [14]. This test maintains better control of the false-positive rate than the test proposed by Egger at al [14].

Results

Study characteristics

Thirty-five studies [15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49] reported 45 independent trials included in one or more of the three comparisons (Table 1). The studies were conducted in the US (n = 20), Europe (n = 13), and Australasia (n = 2). The studies included between 133 and 12,734 participants and were published between 2001 and 2020. Eight studies were judged to be at high risk of bias [16, 19, 33, 34, 42, 43, 45, 46].

Table 1 Characteristics of 35 included studies
Full size table

Results of syntheses

Comparison 1 - Postal vs. electronic questionnaire

Twenty-seven trials (66,118 participants) evaluated the effect on questionnaire response of postal administration compared with electronic. [15,16,17,18,19,20, 23,24,25,26,27,28, 31,32,33,34,35,36, 38,39,40,41, 43, 44, 46,47,48] The odds of response were increased by over half (OR 1.76; 95% CI 1.34 to 2.32) using a postal questionnaire when compared with an electronic one (Fig. 1). There was considerable heterogeneity between the trial results (I-squared = 98%), but most of the studies showed response was greater with postal questionnaires than with electronic questionnaires, and the high I-squared is due to differences in the size of the benefit for postal questionnaires, rather than being due to an even spread of results between those favouring postal and those favouring electronic questionnaires.

Fig. 1
figure 1

Effect on response of mode of administration

Full size image

Comparison 2 - electronic questionnaire only vs. choice (postal or electronic)

Ten trials (39,523 participants) evaluated the effect on questionnaire response of providing a choice of response mode (postal or electronic) compared to an electronic questionnaire only [20, 21, 27, 29, 30, 35, 37, 40, 42, 45]. The odds of response were increased by over half when providing a choice of response mode (OR 1.63; 95% CI 1.18 to 2.26; Fig. 2). There was considerable heterogeneity between the trial results (I-squared = 97.1%), but again, most of the studies favoured giving people the choice of response mode rather than electronic questionnaire only, and the high I-squared is due to differences in the size of the benefit for choice, rather than being due to an even spread of results between those favouring choice and those favouring electronic only.

Fig. 2
figure 2

Effect on response of choice of response mode compared with electronic only

Full size image

Comparison 3 - choice (electronic or postal) vs. postal only

Eight trials (20,909 participants) evaluated the effect of providing a choice of response mode (electronic or postal) compared to postal response only [20, 22, 27, 29, 34, 35, 40, 49]. There was no evidence for an effect on response of providing a choice (OR 0.94; 95% CI 0.86 to 1.02; Fig. 3). There was moderate heterogeneity among the trial results (I-squared = 50.9%).

Fig. 3
figure 3

Effect on response of choice of response mode compared with postal only

Full size image

Results of subgroup analyses

Table 2 presents the results of subgroup analyses according to the two study-level factors (forest plots of these subgroup analyses are included in supplementary figures).

Table 2 Results of subgroup analyses of according to two study-level factors
Full size table

Comparison 1 - postal vs. electronic questionnaire

Year of publication

A third of studies were published before 2012 [15,16,17, 23, 24, 33, 35, 40, 47, 48]. In this subgroup of studies the odds of response were 85% greater (OR 1.85; 95% CI 1.12 to 3.06) with a postal questionnaire compared with an electronic one. In the subgroup of studies published after 2012 the effect was lower (OR 1.70; 1.19 to 2.43), consistent with our concern (Sect. 2.4.1) that higher household access to a computer from 2012 may have improved preference for electronic questionnaires, however the statistical test of homogeneity of the pooled effects in these two subgroups was not significant (p = 0.788), indicating no evidence from these studies for different effects by year of study (Supplementary Fig. 1a).

Risk of bias

Seven of the trials [16, 17, 26, 33, 34, 43, 46] were judged to be at high risk of bias and for these trials the odds of response were more than tripled (OR 3.24; 95% CI 1.68 to 6.25) using a postal questionnaire when compared with an electronic one. There was considerable heterogeneity between the trial results (I-squared = 99%).

When only the 20 trials deemed to be at low risk of bias were synthesised, the odds of response were increased by two-fifths (OR 1.43; 95% CI 1.08 to 1.89). There was also considerable heterogeneity between these trial results (I-squared = 96.8%).

The statistical test of homogeneity of the pooled effects in these two subgroups (p = 0.025) provides some evidence for greater effect estimates in studies at high risk of bias (Supplementary Fig. 1b). Our best estimate of the effect on response of mode of administration is hence from a synthesis of the studies at low risk of bias (OR 1.43; 95% CI 1.08 to 1.89). Results overall were thus confounded by risk of bias, but this did not explain the between study heterogeneity.

Comparison 2 - electronic questionnaire only vs. choice (postal or electronic)

Year of study

Half of studies were published before 2012 [35, 40, 42, 45]. In this subgroup of studies there was no evidence for an effect on response of providing a postal response option (OR 1.22; 95% CI 0.93 to 1.61). In the subgroup of studies published after 2012 there was evidence for an effect on response of providing a postal response option (OR 2.02; 95% CI 1.30 to 3.13). The statistical test of homogeneity of the pooled effects in these two subgroups was significant (p = 0.057), indicating some evidence from these studies for different effects by year of study (Supplementary Fig. 2a). This apparent preference for a postal response option in studies published after 2012 was counter to our concern (Sect. 2.4.1) that higher household access to a computer from 2012 would improve preference for electronic questionnaires. There was considerable heterogeneity between the trial results (I-squared = 98.2%), but most of the studies favoured giving people the choice of response mode rather than electronic questionnaire only, and the high I-squared is due to differences in the size of the benefit for choice, rather than being due to an even spread of results between those favouring choice and those favouring electronic only.

Risk of bias

Two of the trials were judged to be at high risk of bias [42, 45]. There was no evidence for an effect on response of a postal option in these studies (OR 1.08; 95% CI 0.43 to 2.71). When only the 8 trials deemed to be at low risk of bias were synthesised, there was evidence that the odds of response were increased when providing a postal response option (OR 1.77; 95% CI 1.23 to 2.55). There was considerable heterogeneity between these trial results (I-squared = 97.7%). The statistical test of homogeneity of the pooled effects in these two subgroups (p = 0.326) provides no evidence for different effects by risk of bias (Supplementary Fig. 2b). Our best estimate of the effect on response of providing a postal response option is hence from a synthesis of all of these studies (OR 1.63; 95% CI 1.18 to 2.26).

Comparison 3 - choice (electronic or postal) vs. postal questionnaire only

Year of study

In the subgroup of studies published before 2012 there was very weak evidence that the odds of response were lower with an electronic option (OR 0.85; 0.73 to 0.98), whereas in studies published after 2012 there was no evidence for a difference between an electronic option and postal only – perhaps due to electronic methods being more acceptable with increased computer access. The results in both subgroups were more homogeneous (I-squared = 48.5% and 7.0% respectively). The statistical test of homogeneity of the pooled effects in these two subgroups (p = 0.04) provides some evidence for different effects by year of study (Supplementary Fig. 3a). If we consider the most recent trials to better represent the situation today (i.e., greater access to computers than prior to 2012), then our best estimate of the effect on response of providing an electronic response option is from a synthesis of the studies published after 2012 (OR 1.01; 95% CI 0.93 to 1.08), i.e., no evidence for an effect.

Risk of bias

There was one study at high risk of bias [34]. Its results were entirely consistent with the results of the seven studies at low risk of bias (the statistical test of homogeneity of the pooled effects in these two subgroups was not significant (p = 0.454), Supplementary Fig. 3b).

Results of assessments of evidence for reporting bias

Comparison 1 - postal vs. electronic questionnaire

There was no evidence for small study effects (Harbord’s modified test p = 0.148).

Comparison 2 - electronic questionnaire only vs. choice (postal or electronic)

There was no evidence for small study effects (Harbord’s modified test p = 0.841).

Comparison 3 - choice (electronic or postal) vs. postal questionnaire only

There was no evidence for small study effects (Harbord’s modified test p = 0.139).

Discussion

General interpretation of the results in the context of other evidence

This study has shown that response to a postal questionnaire is more likely than response to an electronic questionnaire. It has also shown that response is more likely when providing the option for postal response with an electronic questionnaire. It has further shown that providing an electronic response option with a postal questionnaire has no effect on response. Response is thus increased using postal rather than electronic questionnaires.

A previous meta-analysis of 43 mixed-mode surveys from 1996 to 2006 also found paper and postal administration produced greater response than electronic administration [50]. Our result that providing an electronic response option to postal administration does not increase response is consistent with a previous meta-analysis of randomised trials that found that mailed surveys that incorporate a concurrent Web option have significantly lower response rates than those that do not [51].

We suggest two possible reasons for these results:

  • Paper questionnaires are more accessible than electronic questionnaires.

Although access to the Internet increased over the period during which the studies included in this study were conducted [5, 52], a ‘digital divide’ [53] persists in many populations where completion of a paper questionnaire may be possible, but completion of an electronic one may not.

  • Paper questionnaires are more personal than electronic questionnaires.

Personalised materials have been shown to increase response [54]. If participants perceive a paper questionnaire with a return envelope to be more ‘personal’ than a request to go to a website to answer some questions, we should expect a higher response with paper.

Strengths and limitations

The main strengths of this study are that our results are based on syntheses of the results of 45 randomised controlled trials that span two decades, and most of which were judged to be at low risk of bias.

There was, however, considerable heterogeneity between the results of the included studies. Our subgroup analyses did not identify any causes of heterogeneity among study results, but they did reveal confounding of the pooled result for postal versus electronic questionnaires. The unexplained heterogeneity means that we cannot be confident about the magnitude of the effects on response using postal rather than electronic questionnaires. However, from inspection of the forest plots we can be confident about the direction of these effects.

The evidence included in this review addresses ‘unit’ non-response only (i.e., return of questionnaires). ‘Item’ response (i.e., completion of individual questions) may be greater with electronic methods, but this was not addressed in this review and requires investigation in the future.

We assessed evidence for reporting bias using Harbord’s modified test for small-study effects and found no evidence for bias. This test may not be reliable given the substantial heterogeneity between the results of the included trials [55].

Due to the nature of this study (secondary analysis of a published review), there is no pre-registered protocol for the subgroup analyses provided in this study.

Implications for practice, policy, and future research

These results will help researchers and healthcare providers to improve data collection from study participants and patients, helping to maintain study power and reduce bias due to missing data in research studies. In addition to the methods already known to be effective in increasing questionnaire response [8, 56], postal questionnaires should be used in preference to, or offered in addition to, electronic modes as this will help to increase the proportion of participants that responds. It should be noted, however, that the evidence upon which this recommendation is based is from studies published between 2001 and 2020, and this may change in the future as access to the Internet increases and more people become ‘tech-savvy’. Furthermore, we consider that the certainty of the evidence provided in this study is “Moderate”, due to the unexplained heterogeneity between the results of the included studies.

Future research

Evidence on effective data collection in low- and middle-income settings is needed. Research centres such as LSHTM can embed studies within trials (SWATs) in their research in these settings to help to increase the evidence base [57].

Participation rates for epidemiologic studies have been declining [58]. Our study has presented evidence that postal questionnaires are preferable to electronic questionnaires to improve participation, but it does not tell us why. Research is still needed to advance sociological and psychological theories of participation in data collection procedures [59].

Electronic administration provides benefits for researchers over paper administration which have not been addressed by this study: A well-designed Web questionnaire can control skip patterns, check for allowable values and ranges and response consistencies, and it can include instructions and explanations about why a question is being asked [60]. These options could help to improve the completeness and quality of self-administered data collection, maintaining study power, reducing the risk of bias in study results, and saving study resources. Further research into the cost-effectiveness of electronic administration compared with postal administration in different settings will be needed to inform practice [61].