Canadians living below the low income cut-off, non post-secondary graduates and new Canadians were specifically targeted in this study. These populations are notorious for having low response rates;[1–4, 24] therefore, it was particularly important for the SPAACE study to utilise accepted methods of increasing response rates. Hence, randomly selected households were mailed pre-contact letters that were personally addressed and indicated that the researchers were affiliated with Canadian universities and that the study was publicly funded[5]. Potential participants were also referred to a website should they seek further information regarding the study. They were advised that participation was entirely voluntary as it has been shown that stressing personal choice rather than obliging participation results in higher response rates[25]. Further, the letter did not disclose the true purpose of the study, i.e. to assess the prevalence of food allergies, as it was thought that this may introduce selection bias. Households who are actually affected by food allergy may have been more willing to participate, leading to an overrepresentation of those with food allergy and an overestimation of food allergy prevalence[3]. Finally, multiple callbacks were made on difference days and at different times[26].
In addition to the numerous strategies outlined above, we examined the influence of unconditional incentives on the response rate. Based on a predictive model developed by Edwards et al. where the odds of response increased significantly for each $0.01 increase in incentive value up until $5,[9] we chose to use prepaid incentives valued at $5. An increase in response rate, cooperation rate and less conservative cooperation rate of 7.4%, 8.2% and 6.4% respectively were observed in the incentive versus non-incentive group; however, these differences were accompanied by wide CIs, which did not completely exclude very small or even negative values. A larger sample size may have provided smaller CIs making our conclusion more definitive. It was hoped that an increase in response of 15% would be achieved with incentives and although a difference of 15% can be seen in the upper confidence limits of all three response proportions, the width of these CIs makes it unlikely that the true difference is actually this large. Nevertheless, our results suggest that an unconditional incentive likely increases the response rate in our targeted population. Although others have demonstrated the positive effect of incentives on response rates,[6, 7, 9–11, 16, 27] ours was the first to examine unconditional incentives in these vulnerable populations for a telephone survey.
Conversely, some studies have reported that incentives may not be beneficial[1, 12, 28]. Wenemark et al. suggests that the positive impact of incentives may be specific to consumer related research. It is speculated that incentives may insult or annoy potential respondents when the research is health related; that is, subjects may already feel an obligation towards participating in medical research, and offering incentives may induce suspicions of ulterior motives[28].
Some have reported that incentives are more effective among low income or visible minority populations possibly because incentives are of greater value to these individuals[1, 3, 17–19]. Our results do suggest that incentives may be more effective in lower socioeconomic status populations as a higher proportion of respondents in the incentive group were living at or below the low income cut-off. In contrast, the proportion of respondents without a post-secondary education and born outside Canada was comparable between the groups, suggesting that the incentive was not necessarily more effective in these populations. These conclusions are valid only if we assume that the incentive and non-incentive group had similar demographic characteristics due to the random administration of the incentive.
The ability of incentives to increase participation may be explained by the theory of cognitive dissonance. The theory states that a subject will experience a state of aversive arousal referred to as “dissonance” when they have received an incentive and decide to not participate in the survey. This aversive arousal is a result of the inconsistent idea of receiving something for nothing. To reduce this dissonance, the subject may then decide to participate[25]. Biner and Barton argue that this popular theory contains a flaw, and the more appropriate theory is that of equity. Equity theory states that feelings of guilt result when a subject feels they are being overcompensated for their actions. To reduce this guilt, a subject will choose to participate. The major difference between these theories is that in the equity stream, the potential respondent only considers not participating in the survey whereas in dissonance, the subject makes a decision and subsequently changes their mind[29].
A limitation of our study is that we could not determine if the increased response rate in the incentive group improved the validity of our food allergy prevalence estimates. Although there is a gold standard for determining the prevalence of food allergy, the food challenge, it is potentially very dangerous as the individual is given a food to which they may have a life-threatening allergic reaction. Further, performing a food challenge is very labor intensive, time consuming, and costly and for all these reasons, cannot be applied in this nationwide survey. Therefore, for the purposes of this study, there is no gold standard for determining food allergy. If the prevalence of food allergy did differ between the incentive and non-incentive group, we could not determine which prevalence is more accurate. In addition, the prevalence of allergy to specific foods is at most 2% and therefore our sample size would be too small to accurately estimate any difference between the incentive and non-incentive group. Although it is hoped that a higher response rate will decrease response bias providing a more accurate estimate of food allergy prevalence, it is recognized that the increase in response rate achieved with an incentive does not necessarily automatically decrease response bias. It is possible that only a certain type of non-respondent is encouraged to participate when incentives are utilized, resulting in increased bias, despite the increased response rate.
A second limitation is that the incentive group received two different incentives, one from a major food manufacturer and the other from a nationwide coffee shop. Although both incentives were worth $5, subjects may value these compensations differently; grocery store gift cards can provide necessities whereas coffee shop gift cards cannot. Although it would have been of interest to formally compare the effect of each incentive on the response rate, this would have required a total sample size of over 2000 and was therefore not feasible for a pilot study. Nevertheless, response proportions were calculated when stratified by incentive type (Table4) and although no differences were observed, a larger sample size may have revealed a meaningful difference. Another limitation of this study was there was no way to confirm that all households in the incentive group actually received the incentive. It was thought that including such a query may influence the interviewers thereby potentially introducing bias. As well, it should be noted that our population was sampled from a directory of telephone numbers associated with mailable addresses and our conclusions may not apply to household telephone numbers without mailable addresses as their response rates may differ.