Background

Longitudinal cohort studies are important for the understanding of aetiological mechanisms underlying population and individual differences in the incidence of disease and for monitoring social inequalities in health[1, 2]. Selective attrition, however, is a known problem in cohorts as those in disadvantaged socio-economic groups, ethnic minorities, younger and older people and those at greater risk of ill-health are more likely to drop out[3]. This may result the generalisability of findings being limited and estimates of association being biased[4]. Although direct evidence for this is limited with some studies[5] finding evidence of biased estimators with different response rates, while others found the associates were unaffected by selective attrition[6–8].

Overall, therefore, against a background of declining response rates in surveys in the UK[9] and in many cohort studies[10] more focused efforts to prevent attrition are required to ensure the benefit of the findings from cohort studies to public health do not become limited.

The barriers to recruitment and retention of participants in clinical trials are fairly well documented. General distrust of researchers and studies, concerns about research design, the consent process, discordance between lay beliefs and medical practice, patient treatment preferences, uncertainty about outcomes, and additional demands of the trial (e.g. duration of interventions, cost of travel, etc.) are frequently cited reasons for non-participation[11–13]. Similar reasons have recently been cited for non-participation in longitudinal cohort studies[14, 15]. While a number of reviews have reported different ways of improving study participation, as well as the contextual factors that may affect these approaches,[16–18] little is known about the effectiveness of specific retention strategies, which may differ by study design. Reasons for attrition may differ for randomized trials, e.g. random assignment to an unwanted treatment group, and the process of randomization and the types of interventions, which, make it difficult to extrapolate the effectiveness of certain retention methods to cohort studies. Cohort studies are expensive, with the follow-up of high risk groups requiring the most effort and resources, so there is a critical need to identify effective retention strategies. For these reasons, in this review, we have decided to focus exclusively on cohort studies. The main objective of this review was to determine the effectiveness of retention strategies in improving retention rates in prospective population-based cohort studies.

Methods

This review focused on the evaluation of retention strategies in prospective population-based cohort studies with health as an outcome. A population-based cohort was defined as "any well-defined population defined by geographic boundaries, membership or occupation"[19]. Studies were included if there was at least one wave of follow-up data collection in which the participant, or a proxy, was personally contacted by the study, at least one retention method was described and method-specific retention rates were reported. Studies were excluded if they were clinical or non-clinical trials evaluating the effectiveness of treatment regimens or intervention/prevention programmes, non-population-based cohort studies or cohorts with record linkage as the only method of follow-up. Only English language studies were searched and selected in order to reduce potential biases from misinterpretation. Studies which focus solely on locating (e.g. tracing) respondents, although an important activity for cohort maintenance, were not included in this review. Similar reviews of the effect of initial recruitment on subsequent retention have not been considered here, although again there is evidence that significant effort at recruitment may reduce subsequent attrition[20].

The electronic databases Medline, PsycINFO, PsycABSTRACTS, Embase, CINAHL, ISI, AMED and the Cochrane Central Register of Controlled Trials were initially searched for studies published through to June, 2007. The review was updated with additional searches of Medline, PsycINFO, ISI and the Cochrane Central Register of Controlled Trials conducted in November 2008 and in January 2011. Existing reviews were identified by searching the Cochrane library, the Database of Abstracts of Reviews of Effects (DARE) public databases at the Centre for Reviews and Dissemination, and other relevant medical and social science databases, including those produced by the Health Development Agency. Manual searches of bibliographies were also conducted to obtain reports on primary studies that were not retrieved through the electronic search. A list of potential prospective population-based cohort studies was also developed and study investigators were contacted and study websites searched for unpublished and technical reports.

Five terms were used in the electronic search 1) recruitment, 2) retention, 3) attrition, 4) participation, and 5) study design. In most cases a second keyword was included with the main search term, for example, attrition was paired up with any variation of "minimi" (e.g. minimization, minimizing, etc.). Use of the term 'recruitment' enabled identification of cohort studies that may have been missed through use of the retention term only. The search was restricted to include only publications where the two words were within two words of each other (i.e. minimization of attrition, attrition was minimized, etc.). Specific terms were agreed upon by the authors and adapted for each database (Additional File 1). CB conducted the initial appraisal of all titles and abstracts of papers. SH conducted a 20% re-check working independently to ensure that potentially relevant cohort studies or retention evaluations were not missed. Any disagreements were resolved through discussion. Data items to be extracted were agreed by all authors, and a data extraction database containing details of each study was developed by CB. MB independently reviewed all data extracted.

Retention strategies were categorised as 1) incentives to participate (monetary and non-monetary), 2) reminder calls or letters, repeat visits (i.e. more than one visit to schools to follow-up pupils who were not present on previous days of data collection) or repeat questionnaires and alternative modes of data collection and 3) other methods (e.g. method of posting, length of questionnaire). Differences in retention rates across the different retention strategies were examined with Meta-Analyst software[21]. Individual study proportions, that is the number of participants retained from a specific retention method divided by the number of participants approached, and 95% confidence intervals were calculated using random model analysis weighted by sample size and variance[21]. The individual study proportions given in the tables are the additional increase in the proportion of subjects retained from the specified method. Due to the heterogeneity of the methods within and between the RCTs and non-experimental studies meta-analyses were not conducted.

Results

Literature Search

The literature, bibliography, and website searches, together with correspondence with study investigators identified 17 210 papers. As Figure 1 shows, the vast majority of these were excluded because they were not population-based studies or they focused on recruitment rather than retention strategies, leaving 913 potential papers. Two-thirds of these papers were excluded because they had no information on the retention strategies, 30% of which were then excluded as they did not contain information on the evaluation of retention strategies.

Figure 1
figure 1

Flowchart of search methodology.

Full size image

Twenty-eight studies from thirty-two papers, unpublished papers, technical reports, book chapters, and one personal communication were identified as eligible for inclusion in this review[22–53].,[Dudas, e-mail, 14 December 2007] Table 1 provides a description of each study (see Additional File 2 for a more detailed description).

Table 1 Description of Studies Included in Review
Full size table

Of the 28 studies reviewed more than half were conducted in the USA, were postal questionnaires and were conducted with adult cohorts. The majority were less than 10 years old when the retention strategy took place and had less than 10 follow ups.

Of the 28 studies, 11 were RCTs,[22–33] of which 9 focused on the effectiveness of incentives and 2 experimented with the interview length and postal methods; the remaining 17 conducted other types of analyses of the effectiveness of the retention methods used[34–53].,[Dudas, e-mail, 14 December 2007] Some of the retention efforts were conducted in pilot studies[36, 40, 42] and others were trialled after the main attempt data collection had been completed[22, 24, 27–31, 33]. This use of retention methods on sub-populations may have a significant effect on the response rates reported. For example, pilots on hard-to-reach groups or reluctant participants may have low response rates; however, the addition of these participants could improve both the overall response rate and representativeness of the study population.

Incentives

Incentives were evaluated in ten studies,[22, 23, 25–28, 32, 33, 40, 45] the results of which are shown in Table 2. Incentives were associated with an increase in overall retention rates[22, 25–28, 32, 33, 45]. Five studies[23, 25, 26, 32, 40, 45] trialled incentives with all study participants, i.e. incentives were included in the first data collection attempt. The increases associated with the provision of, or increase in the value of incentives, ranged from 2% to 13%. Two studies[25, 26, 32] randomized differing amounts of monetary incentives and results showed that retention was higher in groups that received higher amounts. Two studies[23, 40] examined the effects of non-monetary incentives on retention and found no increase in retention. However, Rudy et al. in a non-experimental study, reported a 79% response rate for those receiving $100 incentive and 66% for those receiving non-cash gifts (X 2 (1,166) p < 0.05)[45]. Retention rates also increased with the value of monetary incentive offered[22, 25–28, 32, 33].

Table 2 Increase in Study Retention Rates for Incentive and Reminder Letters by Data Collection Type
Full size table

An exception to the finding of increased response with greater monetary value was the study by Doody et al. which, found that a $2 bill resulted in a higher retention rate than a $5 cheque[22]. One possible explanation is that in the United States, a $2 bill is rare and may have novelty; alternatively, the higher amount was given as a cheque and the transaction cost of cashing a low-value cheque may have reduced the effect of the incentive[22]. The National Longitudinal Surveys (NLS) found that not all respondents who received cheques or cash cards as incentives used them. Although equivalent cash value were not offered to provide a direct comparison, it is likely that while reducing the overall cost of incentives to the study,[27, 28] this strategy also reduced the impact of an incentive on the retention rate. There is tentative evidence that providing incentives, particularly upfront to specific groups, e.g. non-responders to previous data collection, may reduce the cost per interview as the cost of the incentive is cheaper than multiple visits or calls to obtain such respondents without an incentive[27, 28].

Reminders, repeat contacts and alternative modes of data collection

The most common approach to improving retention was to write to or call respondents to remind them to complete a questionnaire or take part in an interview; to send additional questionnaires, make repeat calls or visits; or, to offer alternative modes of data collection in an attempt to capture reluctant respondents. Seventeen studies included at least one of these methods,[22, 24, 33–44, 46–53] most including a range of them in a hierarchical fashion, starting with the least labour intensive (i.e. reminders) and ending with the most costly (i.e. alternative modes of data collection). With the exception of one study[42] it was possible to separate out the effect of each specific stage on retention.

Table 2 shows the additional proportion retained after posting reminder letters or postcard following a postal survey, which appeared to increase with number of letters sent. The time between sending out the initial postal questionnaire and the reminder varied by study, but no study evaluated the optimal time between postings.

Ten studies posted questionnaires to participants multiple times,[22, 31, 33, 35–37, 40, 42, 46, 48], [Dudas, e-mail, 14 December 2007] with nine[22, 31, 33, 35–37, 40, 46, 48], [Dudas, e-mail, 14 December 2007] providing retention rates for each posting. Table 3 shows that the additional proportion retained from posting repeat questionnaires appeared to increase with the number posted. Only one study compared the effectiveness of reminder letters with that of repeat questionnaires. Hoffman et al. found that those who received a second questionnaire were much more likely to be retained than those who received only a reminder postcard[40].

Table 3 Increase in Study Retention Rates for Repeat Questionnaires and Alternative Methods of Data Collection by Data Collection Type
Full size table

Ten[34, 35, 38, 41–44, 46, 47, 50–53] of the twenty-eight studies offered alternative data collection modes to participants; seven[34, 35, 38, 42, 46, 47, 50–53] of these studies had already used other retention methods. Table 3 shows that there was an increase in retention with any alternative additional data collection method. The additional retention was highest for face-to-face studies,[41, 43, 44] which conducted the first interview in a central location e.g. a clinic or school and the subsequent modes either followed up with home interviews or postal questionnaires[41, 43, 44]. In these studies, alternative modes of data collection were generally the second stage of the study, which in addition to the convenience of home-based interviews might help to explain the larger average increases in retention[41, 43, 44].

The increase in retention from reminder calls made for postal survey studies was also examined in four studies (data not shown); all of which had already sent reminder letters[34, 37, 46, 50–53]. Reminder calls appeared to have a greater effect on younger age cohorts,[34, 50–53] with an increase of between 10% and 16%, in comparison to increases between 1% and 6% among older cohorts[37, 46, 50–53] retention only.

Two telephone surveys demonstrated the need to make multiple calls to achieve a completed interview (data not shown)[38, 47]. Garcia and colleagues found 62% of participants only required between one and three telephone calls to complete an interview. However, ten or more calls, however, were required to successfully interview 9% of participants[38]. The National Population Health Survey found that less than 15 attempts were needed to conduct 90% of the interviews; however, up to 50 calls were required for the remaining 10%[47].

In a school-based study of multi-ethnic pupils in England, Harding et al., (data not shown) used multiple school visits (i.e. up to 13 additional visits to schools to follow-up pupils who were not present on previous data collections). Retention increased by 26%, 6% and 2% after the second, third and fourth visits respectively[39].

Multiple Methods

Thirteen postal survey studies[22, 24, 33–37, 40, 42, 46, 48–53] in this review used multiple retention methods and eleven of these[22, 24, 33–37, 42, 46, 49–53] had retention rates of more than 70%. This might suggest that the more effort studies put into retaining respondents, including use of multiple methods, the higher the retention rate will be. However, it is important to keep in mind the costs will also be higher. These studies[22, 24, 33–37, 40, 42, 46, 48–53] all began with the cheapest methods, e.g. posting reminder letters, and ended with the most expensive, such as alternative modes of data collection, so that the number of respondents that needed to be captured with each additional method decreased as the costs increased.

Other methods

In addition to the broad approaches described above a few more specific initiatives were tried by some studies. Rimm et al. found that retention was significantly higher for questionnaires sent via certified mail than those sent by other mail types, and for envelopes that were handwritten than those that were not[31]. However, Doody et al. did not find any difference in retention from alternative methods of posting the questionnaire[22]. Kalsbeek and Joens reported some evidence of increased retention by providing personalised information in letters, however as this was combined with non-monetary incentives, determining the true effect of personalization was difficult[23].

In two more detailed experiments, Hoffman et al. found a modest increase in retention if a 4-page questionnaire was used instead of a 16-page questionnaire (p = 0.145);[40] Clarke et al. found that including income questions did not affect retention rates but asking for proxy respondents to complete a cognitive questionnaire about the primary study participants appeared to decreased retention when the main questionnaire was used[36].

Discussion

In the studies reviewed here, incentives were associated with an increase in retention rates, which improved with higher values. Whether cash was the most effective incentive was not clear from studies that compared cash and gifts of similar value. Studies of other methods (i.e. reminder letters or calls and alternative modes of data collection) also demonstrated a benefit, but it was difficult to assess their impact due to a less standardised approach. This is the first-known review of the effect of retention strategies on retention rates specifically focused on population-based cohort studies. It is important to consider the effect of different retention strategies on longitudinal studies specifically as different mechanisms may operate once a participant has been/or expects to be in a study over a period of time.

Strengths and Limitations

A major strength of this review is its extensive systematic search of the literature. The general lack of studies that rigorously evaluate retention methods suggests, therefore, that such evaluations are rarely conducted. A key challenge in this review has been comparing retention rates from studies with different methods of calculating or reporting them. In general, we used the retention rate as reported by the authors but we are aware that different methods of calculation could have been used. For example, among studies with two or more follow-up data collections, some used the number of participants eligible for a specific wave of data collection as the denominator[41, 48] while others used the baseline sample[34, 43, 44, 49]. An additional difficulty was created by the inclusion of new or additional participants between waves (e.g. "booster samples", new members in the household, previous non-responders or drop-outs, or studies adopting a policy of continuous recruitment) and whether they were included or excluded from the denominator.

The majority of studies in this review were conducted in the United States which may limit their generalisability to studies conducted in other countries, which, may have different cultures about participating in research or have different ethical guidelines. We also attempted to examine whether the time between data collections or between baseline data collection and the evaluation wave may have influenced retention. However, due to the small number of studies involved and heterogeneity between them, the findings were not reliable.

Compositional factors such as the gender, age and socioeconomic status of participants and contextual factors such as location of the study, the recruitment methods the tracking methods or other indirect methods such as study loyalty or study publicity may also have influenced the effect of retention methods. Few studies had empirical data or reported these in a systematic way so that evaluation or aggregation could be conducted across studies. However, there is some evidence that additional efforts are required to track and retain vulnerable [54] and/or disadvantaged groups[5]. The differential effect of different retention methods across population groups therefore requires further systematic review.

The use of a narrative approach for this review rather than meta-analysis was due to the relatively small number of studies and the heterogeneity in their methodologies, which limited our ability to quantify effects associated with specific retention methods and carry-out meta-analyses. The findings of this study demonstrate the need for studies to rigorously evaluate their retention methods as well as examine the cost-effectiveness of those methods.

Comparisons of effects of similar methods used in non-cohort studies

In other research, incentives have been shown to have a positive effect on retention rates in postal surveys,[55–57] other study designs, in-person and telephone interviews and online studies,[58–60] and also in the recruitment of study members in longitudinal cohort and cross-sectional studies and clinical trials[61–64]. In a meta-analysis of the methods used in postal studies to increase response and retention rates, the odds of response or retention increased by more than half when monetary incentives were received versus non-monetary incentives[57]. Our review was inconclusive in relation to cash versus gift incentives but it did suggest that cash incentives may have a greater effect than cheques on retention rates. There is some support for this in other studies,[65–71] although they are from health promotion and health care projects rather than epidemiological research studies, and therefore their findings may not be transferable to research studies.

The effect of the timing of incentives on retention rates in our review was unclear. However, the meta-analysis by Edwards et al. showed that prepaid incentives increased response more than conditional incentives (OR = 1.61 [1.36,1.89]) in studies that posted questionnaires[57]. Unconditional incentives were also found to lower attrition in a postal questionnaires[70]. In a recent review of studies that used either face-to-face or telephone interviews, conditional incentives did not increase response compared to unconditional (β = 2.82, SE = 1.78, p > 0.05)[59].

Although there was a general lack of standardised approaches of evaluating other retention methods here, there is support in the literature for a beneficial effect on response and retention rates of reminder methods in trials, cross-sectional, prospective and non-population based cohort studies[57, 72–80].

A recent systematic review of retention methods used for in-person follow-up showed that retention rates increased with the number of methods used,[81] which is supported by our findings. The relative lack of evaluation of retention strategies in cohort studies is possibly linked to funding constraints as well as to the potential threat of compromised retention from employing control arms. The use of sub-studies[22, 27–30] or pilot studies[36, 40, 42] provided useful insights about how retention can be enhanced by the evaluation of methods without compromising retention rates. Olson argued that targeted strategies, such as incentives to non-responders from previous waves of the study, is a cost effective approach to retaining those participants who often drop out of studies[27, 28].

Conclusions

Producing generalisable results is a key objective of cohort studies to ensure that the benefits of research can be applied to a wider population. Researchers are encouraged to ensure that participants are given the opportunity to take part and are not excluded due to socioeconomic disadvantage. Much has been written on the ethics of incentives,[82–84] but there is still a lack of consensus, for example, whether varying incentives amounts should be offered to different sub-samples in a study. There is little ethical discussion about whether repeated attempts to obtain consent to follow-up is perceived as pressure to participate or whether research ethics should be adapted to suit the cultural/socio-economic characteristics of the study population. Due to international differences in the regulation of research, the approach to these issues will invariably vary. There was little mention of these issues in the studies we reviewed.

The cost of evaluation, and the risk to study loyalty among participants, may explain the small number of studies that evaluated retention strategies or examined their cost-effectiveness. Raising awareness of the need for such studies among researchers and funding bodies is important to ensure the longevity and scientific value of cohort studies in the future.