Background

Under-recruitment to randomised controlled trials (RCTs) is a major source of inefficiency in the conduct of applied clinical research [1]. Slow recruitment delays the delivery of research and increases costs by increasing the number of staff and sites or by extending the amount and duration of funding required. If a trial extension is not possible, then under-recruitment will increase the likelihood of failing to detect a clinically relevant intervention effect if it exists, which may prevent patients from benefiting from a potentially efficacious intervention. This problem is extensive in clinical trials research – a review of 73 RCTs funded by the Medical Research Council (MRC) or Health Technology Assessment (HTA) in the UK in 2002–2008 found that almost half of the RCTs did not recruit their originally specified target sample size and nearly half of the RCTs received an extension of some kind [2].

The REstart or STop Antithrombotics Randomised Trial (RESTART, ISRCTN71907627, www.RESTARTtrial.org) is an on-going RCT comparing policies of restarting versus avoiding antiplatelet drugs for secondary prevention after stroke due to intracerebral haemorrhage (ICH), which aimed to recruit 720 participants over 2 years (from May 2013 to May 2015) based on recent epidemiological data [3, 4]. RESTART has been behind its recruitment targets despite implementing as many as possible of the strategies that have been shown to maximise recruitment [3, 5,6,7] including minimising the number of eligibility criteria and maximising the time window for recruiting patients after ICH onset.

The aim of the Promoting Recruitment using Information Management Efficiently (PRIME) trial was to investigate whether the rate of randomisation of patients to RESTART could be increased by means of a complex recruitment intervention applied to sites through which investigators in secondary care, with the support of a recruitment co-ordinator, used electronic patient records held by national stroke audits to identify potentially eligible patients.

Methods/design

Design and setting

PRIME is a closed-cohort, stepped-wedge, cluster randomised trial investigating a complex intervention to boost recruitment in RESTART. PRIME is an embedded recruitment trial within the RESTART parallel-group randomised trial. The stepped-wedge design involved a sequential roll out of the PRIME complex intervention to clusters (active hospital sites in RESTART) over twelve 1-month time periods [8]. The PRIME trial is an example of a closed-cohort, stepped-wedge trial because all sites at the start of the trial were expected to continue until the end of the study and no new sites were to be added [9]. All sites began in the control state (no intervention). The month in which the sites started the PRIME complex intervention was randomly allocated so that groups of sites began the intervention sequentially at equally spaced time intervals (steps) as shown in Fig. 1.

Fig. 1
figure 1

The Promoting Recruitment using Information Management Efficiently (PRIME) stepped-wedge trial design

Full size image

Complex recruitment intervention

The PRIME complex intervention involved a recruitment co-ordinator discussing ways to improve recruitment to RESTART at each site via teleconference with the principal investigator and/or the local team, focussing on the provision of software for each site with instructions on how to extract from national stroke audit data lists of their own patients who were potentially eligible for RESTART. There was then a second teleconference to review progress 6 months later. We sent each site a questionnaire before the recruitment review and then a follow-up questionnaire before the 6-month follow-up review. Full details of the PRIME complex intervention, along with inclusion/exclusion criteria and the sample size calculation can be found in the PRIME protocol publication [10].

We created bespoke stroke audit data exports for PRIME in collaboration with the Scottish Stroke Care Audit (SSCA) and the Sentinel Stroke National Audit Programme (SSNAP) teams, to reflect the RESTART eligibility criteria as closely as possible. We provided sites with instructions on how to produce these exports once they had received their recruitment review. Further details can be found in the PRIME protocol [10].

Each audit database produced two bespoke RESTART audit reports (short and long) which could be run separately. The short reports for both were most likely to have suitable patients for RESTART whilst the long reports widened the criteria, and contained further potentially eligible patients. Once the reports were generated the site staff checked that the patient met all the eligibility criteria, using the patient’s medical records and/or via their general practitioner.

The 6-month follow-up questionnaire asked sites to collect information on the number of patients randomised directly from the audit reports.

Site recruitment

Active hospital sites in RESTART were invited to take part in PRIME in June 2015. There were 109 active hospital sites in the RESTART collaboration in the UK at that time. Twenty-four were excluded: two sites in Northern Ireland (where stroke audit data collection was not consistent); two sites in Scotland and two sites in England that piloted the bespoke stroke audit data exports; 14 sites recently activated to RESTART in 2015 (to ensure that all sites taking part had several months recruiting before receiving the intervention); and the remaining four excluded sites were unsuitable for other reasons. The PRIME recruitment co-ordinator telephoned the local RESTART co-ordinator at each of the remaining 85 sites in ascending order of site identification number to invite them to take part in a ‘recruitment review’, following up by email if required, until the sample size target of 72 sites agreed, as shown in the PRIME complex intervention flow chart (Fig. 2). Sites were not informed about the exact content of the recruitment review or that its timing would be randomly allocated. The recruitment co-ordinator and staff at each participating site remained blinded to the timing of the recruitment review until 2 months before the month allocated for each site’s review, when this had to be revealed in order to organise the review.

Fig. 2
figure 2

The Promoting Recruitment using Information Management Efficiently (PRIME) complex intervention flow chart

Full size image

Randomisation

We used stratified block randomisation to randomly allocate the 72 sites into 12 groups of six sites each, with stratification by hospital location (Scotland vs. England/Wales) to ensure that the proportions of sites with access to each national stroke audit data source were approximately consistent across the 12 groups. A programmer independent of the trial created the computer-generated random allocation sequence and sent this list to the trial data manager who then informed the recruitment co-ordinator at the required times for them to arrange the review.

A timeline cluster diagram is shown in Fig. 3 to clarify the timing of trial processes and blinding.

Fig. 3
figure 3

The Promoting Recruitment using Information Management Efficiently (PRIME) Timeline cluster diagram

Full size image

Statistical methods

Full details of the statistical methodology can be found in the published Statistical Analysis Plan [11]. Briefly, the primary outcome of recruitment rate per site per month was analysed using a negative binomial generalised linear mixed model (GLMM), including the fixed-effect explanatory variables: number of months since start of the embedded PRIME study (represented by a linear term in the model to adjust for any background changes in trial recruitment rate), season (December/January versus all other months), site location (Scotland versus England/Wales), and an indicator variable for whether the complex intervention was due to have been implemented or not (according to the planned randomisation schedule). Site was also included as a random effect in the model.

The primary analysis followed an ‘as-randomised’ intention-to-treat principle, which meant that we analysed the data according to the allocated timing of the recruitment review, rather than the time that the recruitment review actually occurred. In addition, for the primary analysis, all 72 sites were included regardless of any site withdrawals or site suspensions or compliance with the PRIME trial procedures. For sites that were suspended (i.e. not permitted to recruit patients due to the principal investigator leaving or another reason), their missing observations were imputed as ‘0’ patients randomised. This was believed to be an appropriate imputation since it is likely, based on the reasons for suspension (e.g. due to the principal investigator being off sick), that many of these sites who were suspended would have failed to randomise any patients if they remained in the trial, and also ‘0’ was by far the most frequently observed number of patients randomised per month. However, to check the robustness of this assumption and the impact of any deviations from the protocol, a per-protocol secondary analysis was performed after excluding any sites experiencing permanent suspensions or sites failing to receive either the initial recruitment review or the 6-month follow-up review.

Results

The PRIME Consolidated Standards of Reporting Trials (CONSORT) flow diagram is shown in Fig. 4. Seventy-two sites were randomised to PRIME; of which 64 (89%) were located in England and Wales and eight (11%) in Scotland. Sixty-four sites (89%) received a recruitment review and 55 (76%) received both a first recruitment review and 6-month follow-up review.

Fig. 4
figure 4

Consolidated Standards of Reporting Trials (CONSORT) 2010 flow diagram

Full size image

Recruitment review

The median delay between the scheduled randomised time of review (first day of the month allocated) and the actual time the recruitment review was delivered was 11.5 days (interquartile range (IQR) 8 to 19.75 days; range 0–71 days). In the 6-month period before any site received the PRIME complex intervention (March 2015 to August 2015), the total number of patients randomised was 0 for 47 sites (65%), 1 for 18 sites (25%), 2 for 6 sites (8%), and 3 for 1 site (1%).

Every time the recruitment review was conducted, we recorded whether there were any issues which impacted the PRIME complex intervention delivery. For 10 sites (14%), the sound quality for the teleconference was quite poor and for two sites (3%) it was very poor. This was due to a mixture of reasons including, sites using speaker phones and separate phones being used in the same room causing echo. For nine sites (12%) the principal investigator was absent from the review, and for three recruitment reviews (4%) the research co-ordinator was ill which may have affected delivery. For the 48 remaining sites (67%), we delivered the recruitment review with no issues.

Table 1 shows responses from the pre-recruitment review questionnaire. Eighteen sites (32%) were already using stroke audit data to recruit to RESTART; the most popular information source to identify eligible patients was screening logs (40 sites, 71%).

Table 1 Responses from the pre-recruitment review questionnaires
Full size table

A complete summary of baseline site characteristics and responses to the pre-review questionnaire stratified by randomised group are shown in Additional file 1.

Primary outcome

The change in the cumulative randomisation total (primary outcome) over time is shown in Fig. 5, with shading to indicate the number of sites that had actually received the recruitment review (i.e. started the intervention) up to that point. Separate plots for each of the randomised groups of sites are shown in Additional file 2.

Fig. 5
figure 5

Change in cumulative randomisation total over time, with darker shading indicating more sites receiving the recruitment review

Full size image

Following imputation of zero monthly counts for nine suspended sites, each of the 72 sites contributed 24 months of observations (March 2015 to February 2017) and so the full primary analysis dataset constituted N = 1728 site-months of observations. Seventy-eight site-months (5%) were imputed with zero monthly counts due to site suspensions.

Table 2 shows the results of the primary analysis on the primary outcome. The adjusted rate ratio for the primary outcome (PRIME complex intervention versus control condition) was estimated to be 1.06 (95% confidence interval 0.55 to 2.03, p = 0.87) after adjusting for site, December/January months, site location (Scotland versus England and Wales), and time in months since the start of the embedded study.

Table 2 Negative-binomial generalised linear mixed model (GLMM) results for the primary analysis of the primary outcome (N = 1728, 72 sites)
Full size table

Per-protocol analysis

Nine sites were permanently suspended during the PRIME study period and, therefore, these sites were excluded from the per-protocol analysis, as were 11 other sites who, for whatever reason, did not receive either a first recruitment review or a 6-month review. After restricting the analysis to the per-protocol population of sites, the adjusted rate ratio became 1.01 (0.50 to 2.04, p = 0.98).

Sensitivity analyses

The results were very similar and conclusions unchanged after performing other pre-specified sensitivity analyses (see Additional file 3). In particular, after restricting the analysis to the intervention roll-out period only (September 2015 to July 2016), the adjusted rate ratio became 1.07 (0.54 to 2.13, p = 0.84). Fitting a second-order spline function to model any secular changes over time improved the precision of estimation and reduced the width of the confidence interval slightly, but there was no change to the conclusions (RR 1.06, 95% CI 0.58 to 1.92, p = 0.86).

Follow-up questionnaires

Fifty-one out of 72 sites (71%) returned the 6-month follow-up questionnaire, and 30 of these (59%) were known to have generated and used the audit data extract reports.

Out of the 30 sites using the audit reports, five sites (17%) ‘strongly agreed’ that the reports were useful, 15 sites ‘agreed’ that the reports were useful (50%), eight sites were ‘neutral’ (27%), and two sites ‘disagreed’ that the reports were useful.

Sixteen sites (53%) indicated that they either had a problem running or using the reports or listed disadvantages to using the reports. The most common disadvantage given was lack of time (seven sites, 23%); and a further three sites (10%) had problems with resource issues: for example, getting consultants or other staff to run the reports or review medical notes. Five sites (17%) complained that the procedure was not identifying many extra patients beyond those already screened as inpatients. Four sites (13%) specified that the reports were picking up a high number of ineligible patients and patients who had died. Two other sites (7%) were concerned about accuracy of the reports, indicating that ‘patients who are already in the trial are not in the list’, and ‘[some patients] weren’t a bleed at all’.

One site that used the audit reports made a suggestion for improvement: they wrote that they would have liked to have had ‘more information regarding understanding the report once produced, and how to link ID numbers to actual patients’.

Twenty-one sites answered ‘no’ to the question of whether they extracted and used the audit data extract reports. The most common barrier cited was resource/staffing/time issues (10/21, 48%); followed by use of other screening methods to identify RESTART patients (6/21, 29%); and problems running the reports or accessing the SSNAP database (6/21, 29%).

Where sites did create audit reports, these were run to an average of 3 years back (range 0.5–6 years) (see Table 3). Seventeen sites (68%) ran the audit reports once; seven sites ran the reports twice (28%); and one site ran the reports three times.

Table 3 Summary statistics for non-categorical responses to questions in the 6-month post-recruitment review questionnaire. Sites used short, long, or both types of bespoke RESTART audit reports
Full size table

Summary statistics for the number of patients identified by the reports, numbers invited back to clinic and numbers responding are shown in Table 3. Sites had the option to record their answers split by audit report (i.e. short and long), or if preferred, they had the option to record the combined total using the ‘Both’ category.

The number of eligible patients invited to a screening visit, declining to come to a screening visit, and numbers randomised is shown in Table 4.

Table 4 Number of patients reported to have come back to clinic and numbers randomised from the audit reports. Sites used short, long, or both types of bespoke RESTART audit reports
Full size table

Under the ‘Short’ report category, two sites each reported one patient coming back to clinic; both patients were randomised. Under the ‘Long’ report category, one site reported a single patient coming back to clinic, but the patient was not randomised by the time of the 6-month questionnaire. Under the ‘Both’ category, the site with two patients coming back to clinic randomised one of them, and the site with three patients coming back to clinic managed to randomise all three of them. Therefore, the total number of patients randomised was six (half of whom were randomised through a single site). Although it should be noted, for this particular site, the data collected on the 6-month questionnaire was not from the bespoke audit data reports – they extracted a list of all ICH patients from the national audit data.

Discussion

We found insufficient evidence that a complex intervention involving a recruitment review and using audit reports to search electronic records had a substantial impact on increasing recruitment to the RESTART trial. The point estimate of the rate ratio was close to 1 in the primary analysis and in all sensitivity analyses, including the per-protocol analysis. The corresponding 95% confidence limits indicate that a large intervention effect of rate ratio 2 or above is unlikely. Although most respondents (67%) to the 6-month questionnaire agreed that the audit reports were useful, only six patients in total were reported to have been randomised from the audit reports.

Since a detailed protocol was used to ensure fidelity to the intervention; and since 89% of sites received at least the first recruitment review and over 75% of sites received both reviews, the lack of intervention effect was unlikely to be due to failure to implement the intervention successfully. Indeed, even after removing sites that did not follow the protocol, no significant intervention effect was found.

There may be several reasons for the lack of evidence of an intervention effect. First, many sites reported a lack of time and resources to sufficiently search the audit data and use the bespoke audit reports. These issues were also raised as disadvantages by those sites who ran the audit reports. Secondly, it could be that the complex intervention itself was simply not effective. Our secondary outcome data suggested that the median percentage of patients identified by the audit reports that were eligible was low: less than 5%. Also the complex intervention did not include any face-to-face contact with the site, which may have reduced its effectiveness. The Study Within a Trial-1 (SWAT-1) study provided evidence of the effectiveness of an intervention involving a face-to-face site visit combined with scheduled meeting on recruitment rates in a multi-centre randomised trial. However, the findings of this study should be interpreted with caution due to the relatively weak study design (it was a non-randomised before-and-after study), and hence the potential for confounding bias [12].

Besides targeting the local site teams involved in recruitment, interventions could also be devised to target potentially eligible patients directly. Interestingly, a text-message intervention to potentially eligible participants stating that there was only a limited number of places left to join the ‘txt2stop’ smoking cessation trial was found to be effective in increasing recruitment rates compared to a text message with no such scarcity message content [13]. More recently, a cross-factorial, embedded RCT design within the EQUIP trial was used to investigate the effect of an intervention involving advertising patient and public involvement to boost recruitment rates. However, the authors found that the intervention had no significant effect on recruitment rates [14].

Strengths and limitations

A strength of our embedded recruitment trial is that it has a cluster randomised design. Stepped-wedge or parallel-group cluster randomised designs are preferable to alternative non-randomised designs, such as before-and-after studies, because they are less susceptible to confounding bias due to temporal trends [15]. In cases where the potential harms or burden due to an intervention are known to be low, then the stepped-wedge design should be considered because all clusters receive a potentially efficacious intervention, and the trial may be more efficient compared to a parallel-group cluster trial. For recruitment trials embedded in multi-centre trials, it may be expected that there is a high level of heterogeneity in recruitment rates between sites due to substantial differences in site size and, therefore, a stepped-wedge trial design may confer greater power.

A limitation of our study is that the PRIME complex intervention was only implemented by a single recruitment co-ordinator which may limit the generalisability of results to other trials and settings. Even so, a detailed protocol was followed, so it is unlikely that this was a major limitation. The most important factor limiting generalisability was the fact that PRIME was embedded within a single host trial. If the intervention was implemented in other trials and settings the results may have been different.

In contrast, the START programme [16] was designed to test two recruitment interventions consisting of (1) optimised patient information sheets and (2) multimedia approaches to encourage patient involvement in research, across multiple host trials, which has provided more generalisable results compared to confining a recruitment intervention to a single host trial. The START investigators showed that an optimised patient information sheet did not significantly improve recruitment or retention rates into the REFORM trial [17].

A second but less important limitation relates to the 6-month follow-up questionnaire results. There was a limited response rate from sites to the first page of the 6-month questionnaire asking for number of patients randomised from the audit reports, which suggests that there may have been randomisations that resulted from the audit reports that were not recorded. Moreover, some sites reported that they were still working through the lists at the time of the 6-month review. Therefore, the self-reported results of the numbers randomised from the audit reports may only represent partial information. It should also be noted that for at least a few sites, the numbers given were from other lists of patients (apart from the audit reports) and some sites accessed the audit data directly without using the bespoke audit reports, albeit this could still be regarded as an indirect consequence of the intervention.

Implications for further research

Although our study showed no significant effect of the PRIME complex intervention, this does not necessarily mean that similar recruitment interventions involving searching electronic patient records will not be effective in other trials or settings. Indeed, in primary care, using electronic patient records has been shown to boost recruitment to ongoing RCTs [18, 19]. Further work is needed to establish the effectiveness of using electronic patient records in secondary care. Alternative ways to increase recruitment may also need to be investigated, particularly those which minimise impact on clinician time and resources.

Conclusions

We found insufficient evidence that the PRIME complex intervention is effective in increasing the recruitment rate to RESTART, despite the intervention being successfully implemented in the majority of sites. Further research is needed into methods to boost recruitment in trials in secondary care.

Trial registration

PRIME was registered with the Northern Ireland Hub for Trials Methodology Research SWAT repository (SWAT22, http://bit.ly/2a4n7Yb) on 23 December 2015 and was submitted to the Online Resource for Recruitment research in Clinical trials (ORRCA, www.orrca.org.uk).

Trial status

The host trial, RESTART, aimed to recruit 720 participants over 2 years, from May 2013 to May 2015 based on recent epidemiological data [3, 4]. Due to under-recruitment, RESTART’s funder granted an extension of recruitment until 31 May 2018. The trial end date is February 2019. PRIME’s first recruitment review took place in September 2015 and the last review completed in August 2016. The first 6-month follow-up review took place in March 2016 and the last one completed in February 2017. After the final 6-month follow-up was received there was a period of 3 months to analyse the data, meaning that PRIME completed in May 2017.