Design, planning and implementation lessons learnt from a surgical multi-centre randomised controlled trial
Increasingly, pragmatic randomised controlled trials are being used to evaluate surgical interventions, although they present particular difficulties in regards to recruitment and retention.
Procedures and processes related to implementation of a multi-centre pragmatic surgical randomised controlled trial are discussed. In this surgical trial, forecasting of consent rates based on similar trials and micro-costing of study activities with research partners were undertaken and a video was produced targeting recruiting staff with the aim of aiding recruitment. The baseline assessments were reviewed to ensure the timing did not impact on the outcome. Attrition due to procedure waiting time was monitored and data were triangulated for the primary outcome to ensure adequate follow-up data.
Forecasting and costing ensured that the recruitment window was of adequate length and adequate resource was available for study procedures at multiple clinics in each hospital. Recruiting staff found the recruitment video useful. The comparison of patient-reported data collected prior to randomisation and prior to treatment provided confidence in the baseline data. Knowledge of participant dropout due to delays in treatment meant we were able to increase the recruitment target in a timely fashion, and along with the triangulation of data sources, this ensured adequate follow-up of randomised participants.
This paper provides a range of evidence-based and experience-based approaches which, collectively, resulted in meeting our study objectives and from which lessons may be transferable.
ISRCTN, ISRCTN41394716. Registered on 10 May 2012.
UKCRN Study ID: 12486.
Activity Capture And Attribution Tool
Clinical Commissioning Group
Data Monitoring and Ethics Committee
Euroqol - 5 dimensions - 5 levels questionnaire
Haemorrhoidal artery ligation
Health Technology Assessment
Health-related quality of life
Haemorrhoidal artery ligation versus rubber band ligation for haemorrhoids trial
Local Research Network
National Health Service
National Institute for Health Research
Research and development
Rubber band ligation
Randomised controlled trial
Studies within a trial
Whole time equivalent
Randomised controlled trials (RCTs) are becoming more widely used to assess surgical interventions despite historical resistance [1, 2, 3]. However, a review across surgical specialities showed over 20% (81/395) of trials are prematurely discontinued , with poor recruitment being the principal reason (36/81); these discontinued trials recruited 15,626 participants. The discontinuation of trials results in considerable wasted investment and at best a less precise answer to the research question [5, 6, 7]. Poor recruitment and retention can lead to withdrawal of funding to complete the trial, which has financial and ethical implications [5, 6, 8].
McCulloch and colleagues  identified different classes of surgical trials with different levels of risk in terms of successful project completion: type 1 trials compare medical management in surgery; type 2 compare surgical techniques; and type 3 compare surgical and non-surgical treatments. Type 3 trials are particularly prone to a lack of clinician and patient equipoise . Recruitment to trials with treatments of differing intensity are often poor [1, 9], with randomized-to-screened ratios of 1:16 documented .
The most common patient-reported reasons for non-entry into surgical RCTs are treatment preference or dislike of randomisation [11, 12] and, where treatments are markedly different, there is increased likelihood of patients or clinicians declaring a preference. In addition, recruiting clinicians often struggle to explain concepts such as randomisation and equipoise [13, 14, 15] and with the amount and clarity of information provided during the consent process . There have been several articles looking at strategies to improve recruitment and retention in trials [5, 8, 13, 16, 17, 18, 19, 20] and evidence for successful interventions is limited. Qualitative work alongside surgical trials can identify particular issues around recruitment and can train staff to address the absence of equipoise and other issues .
The time waited from consent to surgery is a common reason for attrition [21, 22] and type 3 trials may lead to a greater difference in waiting times between treatment groups than types 1 or 2. With increasing waiting times a problem for some health systems , this should be a consideration in surgical trial design. Baseline measures such as health-related quality of life (HRQoL) often change over time, meaning long waiting times between consent and surgery are a potential source of bias. If baseline measures are taken on the day of surgery there is a possibility that the measures could be affected by the knowledge of the treatment allocation . The clinically intuitive timing for follow-up measures is a timepoint relative to the day of surgery, whereas the scientifically desirable timing is a timepoint relative to the day of randomisation, although there is some evidence that this makes little difference to the reported outcomes .
Costing the resource required to deliver RCTs is an important factor in their success. Published workload models for organisations tend to use accrual data, acuity, or a points scale to estimate the research nurses and/or clinical trial administrator/co-ordinator resource needed to implement an RCT [6, 7, 8, 9, 10, 11]. There is no consensus on which model best evaluates workload in clinical research infrastructure . Systems that reimburse research infrastructure based on accrual data focus on the target accrual compared to the number of whole-time equivalents (WTEs), often without accounting for screen failures, query resolution, long-term follow up, participant attrition, or the complexity of the research protocol. They are criticised for over-simplicity and implicated in staff burnout and poor quality standards [12, 13]. This paper presents lessons from the Haemorrhoidal artery ligation (HAL) versus rubber band ligation (RBL) for haemorrhoids (HubBLe) trial [26, 27, 28], a type 3, multi-centre, surgical RCT, to support the implementation of future studies.
Summary of trial design and procedures
The aim of the HubBLe trial [26, 27, 28] was to establish the clinical effectiveness and cost effectiveness of HAL compared with RBL in the treatment of people with symptomatic second or third degree haemorrhoids. Both treatments are recommended for the treatment of haemorrhoids [29, 30, 31, 32]. The trial was a pragmatic, multi-centre, parallel group RCT involving 18 National Health Service (NHS) hospitals in England and Scotland. After consent, participants were individually randomised to HAL or RBL in equal proportions at all centres using a web-based randomisation system. Participants were followed up at 1 day, 7 days and 21 days, 6 weeks, and 12 months post-procedure. Full details of the trial methods can be found elsewhere [26, 27, 28]; here, we focus on methods aimed at improving participant recruitment and retention to achieve a valid data set.
Methods adopted to meet the recruitment target
Eligibility criteria were broad in order to assure a large pool of patients from which to recruit, whilst ensuring patients were suitable for both procedures. HubBLe can be considered a type 3 trial in which medical management is compared with a surgical intervention [1, 9]: HAL was a procedure undertaken in theatre under general anaesthetic; RBL is a less intensive intervention, typically undertaken as an outpatient procedure, and is often delivered by non-surgeons. A key reason for under-recruitment in RCTs is over-optimism at the trial planning stage regarding how many people who are offered participation in the trial will consent and randomise [33, 34, 35, 36].
Many patients had to be screened for one patient to consent
Time spent per patient screened
Coverage and rationalisation
There were an estimated two eligible patients available per clinic. With an estimate of four surgeons involved at every centre and an estimate of four clinics per centre per week, we recognised the challenge for recruiting research nurses to be available at all clinics with potentially eligible patients (coverage). Given the anticipated screening-randomised-ratio, it was imperative that as many potentially eligible patients were screened as possible. Research nurses worked with clinical teams to corral potentially eligible patients into particular clinics, especially where multiple surgeons share a waiting room. The research nurse could then use their time more efficiently with a view to minimising the number of unscreened patients (rationalisation).
Attribution and reimbursement of costs
Costing of research nurse time per centre
Year 1 (recruitment and follow-up):
Research nurse to screen and recruit patients (recruitment activity).
Research costs (research acitivty): 0.3 whole time equivalent (WTE) of a research nurse for the 12 month recruitment period for research activity.
Undertake data collection for the research, data entry, monitoring and meeting attendance (research activity).
Service support costs (recruitment activity): 0.7 WTE - We looked the LRNs to make up a full time post (based on our assumption about recruitment activity).
Year 2 (follow-up only):
12 month follow-up (a half hour telephone interview plus data entry) and closeout visit, plus support for any monitoring and audit activity required.
Research costs: £300 per participant recruited
In addition to ensuring sufficient recruitment capacity for the trial, the team developed a recruitment video  based on the ProtecT trial team’s work on the explanation of randomisation and equipoise . We interviewed the local ProtecT trial team, two research nurses and a consultant involved in recruitment, about their recruitment experiences and narrated the film to highlight the general principles of equipoise and randomisation and how this specifically related to HubBLe. A recent systematic review of training interventions for trial recruiters  identified six trials [48, 49, 50, 51, 52, 53] that had employed a video as part of a face-to-face workshop for that purpose, but there do not seem to be any training programmes solely using videos to aid recruiters. This was seen to be a low-cost method that could be referred to by recruiters as many times as they wished.
Monitoring of waiting times
The duration between randomisation and treatment was monitored in the trial as we knew that there can be significant issues with waiting times for non-urgent surgeries, and that this could affect dropout rates and the intention-to-treat analysis. Whilst RBL is a simple procedure, which is often done on the day of randomisation, HAL is more intensive, is performed under general anaesthetic and requires a theatre slot to be booked. These conditions created the potential for differential participant attrition, a potential source of bias in our analysis [54, 55]. During the trial one of the centres stopped completing non-urgent surgery, which included our procedures. This has been shown to be a continuing issue for the NHS with one Clinical Commissioning Group (CCG) suspending non-urgent surgery to make financial savings prior to the end of the financial year in 2017 [56, 57]. Due to the long waiting times, for the HAL procedure in particular, and the cancellation of non-urgent procedures at one site, the dropout rate prior to the procedure was higher than anticipated. To account for this observed attrition the recruitment target was increased to 370, during the study, in order to achieve the sample of 350 treated, followed up and analysed participants.
Changes in baseline health state post-randomisation, pre-surgery
Three months into recruitment the baseline data collection was changed to the day of procedure, rather than at randomisation. This was because there was substantial between-site variation in surgical wait times, and a difference in wait times for the two treatments, meaning that scores for change in patient-reported outcome at follow up may have reflected time periods substantially greater than intended, especially in the HAL arm. The risk of bias introduced by anchoring follow up to the time of surgery, rather than the point of randomisation, is theoretical and not supported by empirical evidence .
Six months after this change a Data Management and Ethics Committee (DMEC) member suggested that patient-reported outcomes can be affected by the knowledge of their allocation. Since baseline data collection took place on the day of surgery, most patients already knew their allocation by this point; the concern was that perceived pain and HRQoL may differ between the groups due to expectation bias at this time , even though no procedure had yet taken place. The trial statistician reported that the early data did indeed support this hypothesis, in particular with higher self-reported symptoms in the HAL arm. As a result of this, we added a questionnaire to be completed before randomisation as well as at baseline on the day of surgery if the two were more than 1 week apart.
Methods to ensure a valid primary outcome data set
Three sources of primary data collection
Sometimes, in assessing an outcome, using only one data source may be unreliable and data source triangulation is necessary . Our primary objective was to compare patient-reported symptom recurrence at 12 months following the procedure. Recurrence was defined using a simple dichotomous outcome derived from a previously published systematic review . Patients were asked “At the moment, do you feel your symptoms are: cured or improved compared with before starting treatment; or unchanged or worse compared with before starting treatment.” We also asked patients whether (and which) procedures they had undergone for their haemorrhoids, further to trial treatment, since symptoms may only have resolved as a result of further intervention, and supplemented this with treatments as determined from their hospital notes and general practitioner (GP) in order to minimise attrition and recall bias. Finally, we reviewed adverse events and hospitalisations to identify participants that had ongoing symptoms consistent with persistent or recurrent haemorrhoids (e.g. persistent bleeding) that had not been treated.
Our early funding discussions with sites reduced delays in site set-up prior to the start of recruitment, and the lead site even started 1 month earlier than anticipated. Where sites agreed to our proposal, a full-time research nurse was dedicated to HubBLe during the recruitment period. Sites that exceeded their target recruitment (1, 2, 6 and 9) had a named research nurse responsible for the trial, as did sites 4, 7, and 8, though they did not recruit to target. An informal observation was that at sites where research nurses had less time to work on the trial, recruitment and the non-recruited data collection were generally poorer.
Of the 969 patients screened, 198 were not eligible (including the 2 patients that were withdrawn); the majority of these patients were not approached as clinical note review identified the exclusion criteria. The approximate randomised-to-screened ratio in the trial was 5:13; we therefore needed to formally screen (approximately) 13 people for every 5 randomised. This may underestimate the number of individuals screened, as the recording of data from non-recruited patients can be poor in clinical trials as the focus is on recruited participants.
Reasons for non-enrolment to the trial
Patient not approached
Patient did not attend appointment/uncontactable
Patient preference for RBL
Patient preference for HAL
Patient did not want any intervention or treatment
Patient preference for other surgery
Patient preference for immediate treatment
Patient preference related to general anaesthetic
Patient unsure or declined (no further reason given)
Although we did not assess the impact of the video on recruitment in any formal or structured sense, recruiting staff fed back that thinking about equipoise was very helpful, and that they found it easier to describe the two treatments after watching the video. Particular key points that were highlighted as helpful from the video were that it expressed the uncertainty of the effectiveness of each treatment; gave a similar amount of time discussing each treatment arm and avoid loaded statements that may communicate an unconscious bias for one treatment over another; and it provided the opportunity to check the patient’s understanding as you go.
Withdrawals and waiting times
Reason for withdrawal (reasons for withdrawal from treatment are indicated under “Prior to treatment”)
Reason for withdrawal
N = 24
N = 11
Prior to treatment
Found to be ineligible after randomisation
Participant withdrew consent
Lost to follow up prior to procedure
Symptoms resolved/treated elsewhere
Ineligible at time of procedure
Participant withdrew consent
Changes in symptoms between randomisation and procedure
Agreement between self-completed measures of symptoms, incontinence, EQ-5D-5 L, and pain pre-randomisation and pre-treatment (baseline)
Mean change (95% agreement limits)
Difference in mean change
Ratio of variances
Haemorrhoid symptom score
0.0 (−3.0, 3.0)
0.1 (−3.0, 3.1)
−0.1 (p = 0.823)
0. 96 (p = 0.864)
−0.01 (− 0.13, 0.11)
−0.00 (− 0.12, 0.11)
−0.01 (p = 0.508)
1.11 (p = 0.691)
Vaizey Faecal incontinence score
−0.1 (−5.6, 5.3)
0.4 (−2.7, 3.5)
−0.5 (p = 0.231)
3.14 (p < 0.001)
0.2 (− 3.8, 4.2)
−0.0 (− 2.5, 2.4)
0.2 (p = 0.479)
2.63 (p < 0.001)
Our primary outcome was recurrence at 1 year post-treatment. This included a patient-reported outcome measure supplemented by a case note review of further treatment and haemorrhoid-related events.
Data sources for recurrence at one year
RBL (N = 176)
HAL (N = 161)
Recurrence at one year (total)
Data from GP and consultant questionnairesa
Consolidated standards of reporting trials (CONSORT) diagram
Statement of findings
The HubBLe trial is a relatively rare example of a surgical trial that recruited to target and maintained adequate participant follow up. The HubBLe team reduced the risk of project failure by addressing four key areas. We increased the chances of recruiting to target with broad eligibility criteria as suggested elsewhere [5, 33, 34, 35, 36]; forecasting recruitment rates based on previous studies [34, 37, 60]; accounting for screen failures in resourcing recruitment activity; and highlighting the issue of equipoise in the training of recruitment staff as proposed by Donovan and colleagues [13, 14, 15]. However, as expected for treatments of differing intensity , patient preference for treatment was still a barrier to recruitment. In addition, we reduced the risk of delays to recruitment, as recommended [42, 43, 44, 45, 46], by having early discussions with sites to secure funding for recruitment activity. As waiting times have been shown to be a barrier to treatment [21, 61], we anticipated differences in the time from randomisation to treatment in each arm , reducing the risk of bias due to differential attrition. We avoided variation in length of follow up between arms by collecting baseline data on the day of surgery in addition to randomisation. We then compared the data to check for the influence of expectation bias on self-report measures, as suggested by Schulz , but found no systematic differences between the two timepoints. In “Primary outcome” we reduced the risk of unreliable data by triangulating across three sources .
Strengths and limitations
This paper shows how a trial can use a battery of evidence-based methods and the collected experience of a clinical trials unit  to achieve study objectives. We were not resourced to conduct qualitative research alongside the trial to understand and address recruitment issues, as is now best practice [19, 63] and our approach was somewhat ad hoc, with systematic evaluation of the strategies not conducted. For instance, formal feedback on the recruitment training video, which was produced without any funding, was not elicited to improve future efforts and screening data may have been incomplete, as is common in RCTs , and time spent screening patients was not monitored so we cannot determine if our costing of recruitment activity was appropriate.
Meaning and application of findings
The data in this paper, such as those on consent rates and attrition prior to treatment, can be added to the reference class for surgical trials and used in future forecasting. Data showing the imbalance in recruitment between trial sites are important: difference in site capability is frequently observed and has implications for trial planning [65, 66, 67, 68, 69], especially in allowing over-recruitment in site contracts to compensate for less able sites.
The paper highlights issues around waiting times for surgery in the UK [25, 70, 71, 72, 73] and how this can differ between arms in type 3 trials , which should be accounted for in the sample size estimation and when deciding on the timing of data collection. Consideration needs to be given to whether baseline data should be collected at randomisation or on the day of treatment ; though our data show there is little difference between timepoints. Decisions on whether follow-up data should be anchored to randomisation or to the trial treatment also need to be made: if HubBLe had anchored follow up to randomization rather than to trial treatment, the time between treatment and follow up would have been greater in the RBL group due to the longer waiting times for HAL, which in turn could have affected the primary outcome of recurrence.
Unanswered questions and future research
It may not be possible to repeat the comparatively generous allocation of service support costs to this trial in the UK due to the subsequent introduction of the Department of Health’s new costing template (the Activity Capture and Attribution Tool, or ACAT) and the UK Clinical Research Facility Network Intensity Tool to cost research nurse activity. Our experience on more recent trials is that these two tools may considerably underestimate the research nurse time necessary to undertake essential research procedures, threatening the success of the recruitment effort and the integrity of research data. Published workload models estimating staff resource for RCTs [74, 75, 76, 77, 78, 79] are often criticised for over-simplicity, and their use can lead to staff burnout and poor implementation [80, 81, 82]. The rise of surgical trainee networks in the UK as a force in recruitment may go some way to compensating for the pressures on costs in public sector research, where trainees can be incentivised and co-ordinated to recruit and follow up study participants .
Stronger evidence for recruitment and retention strategies in RCTs is required to improve trial efficiency and meet trial objectives. Trial Forge  is an initiative set up to address the lack of evidence in trial decision-making, which will go some way to evaluate recruitment and retention strategies that can be used across RCTs. The use of studies within a trial (SWATs) to find evidence for implementation of RCTs is becoming more commonplace and could be used to assess some of the strategies presented in this paper [85, 86, 87].
Recruitment to and retention in trials comparing surgical interventions of different intensity is challenging but achievable. This paper provides a range of evidence-based and experience-based approaches, which collectively resulted in meeting our study objectives and from which lessons may be transferable.
We gratefully acknowledge the participants who took part in the HubBLe trial and all of the NHS staff involved in recruitment, treatment, and data collection for the trial.
KB and DH wrote the first draft of the paper and all authors reviewed and provided input on further drafts. SB, DH, and MB conceived of or designed the trial. KB, LS, and SB were involved in the acquisition of data for the work. MB conducted the statistical analysis for the trial and assisted KB with further analysis specific to this paper. KB and LS produced the figures and tables. All authors helped in the interpretation of the data. All authors were involved in the final approval of the version to be published. All authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
The HubBLe trial was funded by the Health Technology Assessment programme (HTA 10/57/46) and is published in full in the HTA journal series, Volume 20, No. 88. Further information available at: http://www.nets.nihr.ac.uk/projects/hta/105746. This report presents independent research commissioned by the NIHR. The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, Medical Research Council (MRC), Central Commissioning Facility (CCF), NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC), the HTA programme, or the Department of Health.
Ethics approval and consent to participate
The study received ethical approval from NRES Committee Yorkshire & The Humber - South Yorkshire (12/YH/0236). Written informed consent was obtained from all trial participants prior to participation.
Consent for publication
The authors declare that they have no competing interests.
- 7.Marcellus L. Are we missing anything? Pursuing research on attrition. Canadian J Nurs Res. 2004;36(3):82-98.Google Scholar
- 10.Anvari M, Allen C, Marshall J, Armstrong D, Goeree R, Ungar W, et al. A randomized controlled trial of laparoscopic nissen fundoplication versus proton pump inhibitors for treatment of patients with chronic gastroesophageal reflux disease: one-year follow-up. Surg Innov. 2006;13:238–49.PubMedCrossRefGoogle Scholar
- 14.Donovan JL, De Salis I, Toerien M, Paramasivan S, Hamdy FC, Blazeby JM. The intellectual challenges and emotional consequences of equipoise contributed to the fragility of recruitment in six randomized controlled trials. J Clin Epidemiol. 2014;67:912–20. https://doi.org/10.1016/j.jclinepi.2014.03.010.CrossRefPubMedPubMedCentralGoogle Scholar
- 15.Wade J, Donovan JL, Athene Lane J, Neal DE, Hamdy FC. It’s not just what you say, it’s also how you say it: opening the “black box” of informed consent appointments in randomised controlled trials. Soc Sci Med. 2009;68:2018–28. https://doi.org/10.1016/j.socscimed.2009.02.023.CrossRefPubMedGoogle Scholar
- 23.Campbell D. 193,000 NHS patients a month waiting beyond target time for surgery. Guardian. 2017; https://www.theguardian.com/society/2017/jan/13/193000-nhs-patients-a-month-waiting-beyond-target-for-surgery. Accessed 3 Sept 2019.
- 27.Brown S, Tiernan J, Biggs K, Hind D, Shephard N, Bradburn M, et al. The HubBLe trial: haemorrhoidal artery ligation (HAL) versus rubber band ligation (RBL) for symptomatic second- and third-degree haemorrhoids: a multicentre randomised controlled trial and health-economic evaluation. Health Technol Assess. 2016;20:1–150. https://doi.org/10.3310/hta20880.CrossRefPubMedPubMedCentralGoogle Scholar
- 28.Brown SR, Tiernan JP, Watson AJM, Biggs K, Shephard N, Wailoo AJ, et al. Haemorrhoidal artery ligation versus rubber band ligation for the management of symptomatic second-degree and third-degree haemorrhoids (HubBLe): a multicentre, open-label, randomised controlled trial. Lancet. 2016;6736:1–9.Google Scholar
- 29.National Institute for Health and Clinical Excellence (NICE). Haemorrhoids - CKS Clinical Knowledge Summaries. 2016. http://cks.nice.org.uk/haemorrhoids#!topicsummary. Accessed 12 Feb 2016.
- 30.National Institute for Health and Clinical Excellence (NICE). Haemorrhoidal artery ligation | Guidance and guidelines. 2010. https://www.nice.org.uk/guidance/ipg342. Accessed 12 Feb 2016.
- 33.Gorringe JAL. Initial preparations for clinical trials. In: Harris EL, Fitzgerald JD, editors. The Principles and Practice of Clinical Trials. Edinburgh: Churchill Livingstone; 1970. p. 41–6.Google Scholar
- 36.Lasagna L. The pharmaceutical revolution forty years later. Rev Farmacol Clin y Exp. 1984;1:157–61.Google Scholar
- 39.Kahneman D, Tversky A. Intuitive prediction: biases and corrective procedures. TIMS Stud Manag Sci. 1977;12:313–27.Google Scholar
- 41.Simmons T. Attributing the costs of health and social care Research & Development (AcoRD). London: Department of Health; 2012.Google Scholar
- 47.scharrvids. Training video for recruitment to a surgical trial with two interventions of different intensity. 2015. https://www.youtube.com/watch?v=OlJJqCvjBM8&feature=youtu.be. Accessed 3 Sept 2019.
- 48.Bernhard J, Butow P, Aldridge J, Juraskova I, Ribi K, Brown R. Communication about standard treatment options and clinical trials: can we teach doctors new skills to improve patient outcomes? Communication about standard treatment options and clinical trials: can we teach doctors new skills to improve patient…. Psychooncology. 2012;21:1265–74. https://doi.org/10.1002/pon.2044.CrossRefPubMedGoogle Scholar
- 50.Kimmick GG, Peterson BL, Kornblith AB, Mandelblatt J, Johnson JL, Wheeler J, et al. Improving accrual of older persons to cancer treatment trials: a randomized trial comparing an educational intervention with standard information: CALGB 360001. J Clin Oncol. 2005;23:2201–7.PubMedCrossRefGoogle Scholar
- 56.Campbell D. NHS cash crisis in Kent halts non-urgent surgery until April. Guardian. 2017; https://www.theguardian.com/society/2017/feb/02/nhs-cash-crisis-in-kent-halts-non-urgent-surgery-until-april. Accessed 3 Sept 2019.
- 57.Royal College of Surgeons. Kent CCG announces unprecedented suspension of non-urgent surgery. Royal College of Surgeons press release. 2017. https://www.rcseng.ac.uk/news-and-events/media-centre/press-releases/west-kent-ccg-delays/. Accessed 3 Sept 2019.
- 58.Herrett E, Shah AD, Boggon R, Denaxas S, Smeeth L, van Staa T, et al. Completeness and diagnostic validity of recording acute myocardial infarction events in primary care, hospital care, disease registry, and national mortality records: cohort study. BMJ. 2013;346:f2350. https://doi.org/10.1136/bmj.f2350.CrossRefPubMedPubMedCentralGoogle Scholar
- 60.Flyvbjerg B, Skamris Holm M, Buhl S. Underestimating costs in public works projects. APA J. 2002;68:279–95.Google Scholar
- 71.McDonald AM, Knight RC, Campbell MK, Entwistle VA, Grant AM, Cook JA, Elbourne DR, Francis D, Garcia J, Roberts I, Snowdon C. What influences recruitment to randomised controlled trials? A review of trials funded by two UK funding agencies. Trials. 2006;7(1):9.Google Scholar
- 75.Berridge J, Coffey M. Workload measurement. Appl Clin Trials. 2008;6:98–101.Google Scholar
- 77.Fowler D, Thomas C. Protocol acuity scoring as a rational approach to clinical research management. Res Pract. 2003;4:64–71.Google Scholar
- 79.National Cancer Institute. NCI Trial Complexity Elements and Scoring Model (Version 1.2; 16 Apr 2009). http://ctep.cancer.gov/protocolDevelopment/docs/trial_complexity_elements_scoring.doc. Accessed 5 Sept 2019.
- 81.Friesema D. Workload assessment in clinical trials. CALGAB Q Newsl Cancer Leuk Gr B. 2010;19:3–8.Google Scholar
- 86.Bower P, Brueton V, Gamble C, Treweek S, Smith CT, Young B, et al. Interventions to improve recruitment and retention in clinical trials: a survey and workshop to assess current practice and future priorities. Trials. 2014;15:399. https://doi.org/10.1186/1745-6215-15-399.CrossRefPubMedPubMedCentralGoogle Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.