Background

‘There is a peculiar paradox that exists in trial execution - we perform clinical trials to generate evidence to improve patient outcomes; however, we conduct clinical trials like anecdotal medicine: (1) we do what we think works; (2) we rely on experience and judgement and (3) limited data to support best practices.’

Monica Shah, quoted in Gheorghiade et al. [1].

This paper summarises a one-day workshop held in Edinburgh on 10 July 2014 to discuss Trial Forge (www.trialforge.org), an initiative focused on improving randomised trial efficiency and quality. The initiative is aimed at the people who design and run trials, staff at trials units, for example, or clinicians and others who design studies. In this paper, we outline the problem of inefficiency in trials and describe the Trial Forge initiative to improve efficiency. We hope that many of those reading the paper will be interested in contributing to Trial Forge in the future.

Randomised trials (hereafter ‘trials’), especially when brought together in systematic reviews, are regarded as the gold standard for evaluating the effects of healthcare treatments, with thousands of trials and hundreds of systematic reviews reported every year. PubMed has indexed over 370,000 reports of randomised trials; the World Health Organisation’s International Clinical Trials Registry Platform [2] contains over 250,000 trial records, of which, 71,000 are listed as recruiting; and the Cochrane Central Register of Controlled Trials contains more than 800,000 records. Tens of billions of dollars of public and private money are invested globally in trials every year (US $25 billion in the United States alone in 2010 [3]) and the average cost of a trial per participant is estimated to be almost £8,500 in the United Kingdom [4].

Many of these resource are wasted, often because insufficient account is taken of existing evidence when choosing questions to address [5], and results are either not published or poorly reported. Moreover, despite trials being a cornerstone of evidence-based healthcare, the methods and infrastructure for conducting these complex studies are largely evidence free [6]. For example, every trial has to recruit and retain participants, but only a handful of recruitment and retention strategies and interventions are currently supported by high-quality evidence [7, 8]. A recent analysis found that only 55 % of UK National Institute of Health Research and Medical Research Council (MRC) trials (a set of large, relatively well-funded studies in the UK) recruiting between 2002 and 2008 met their recruitment targets [9]. The same study found that extensions are common, with 45 % of trials needing at least one funding extension, although only 55 % of these then go on to meet their recruitment targets. Furthermore, although data collection is central to trials and can consume a large proportion of trial resources, researchers often collect more data than they are able to analyse and publish [10]. Indeed, there is a dearth of research into the optimal methods for data collection and data management [11]. This is a different problem from selective reporting, where bias is introduced through the way outcomes are selected and presented in trial reports, especially for harms [12]. Vera-Badillo and colleagues called this type of bias ‘spin’ [13].

As a consequence, the most appropriate methods are not always chosen when trials are designed, leading to trial management and delivery problems later. Indeed, poor design decisions may do more than make a trial difficult to deliver; they may mean that any eventual findings will be of limited value. This could be because, for example, the comparator used renders the trial clinically irrelevant [14], the outcome measures are not relevant to those making treatment decisions [15], or the patients involved do not represent the majority of patients with the condition of interest [16]. The patients, health professionals, and policymakers who look to systematic reviews of trials for help in their decision making are often frustrated to find that the questions addressed by researchers do not reflect clinical decision making needs (a failure of prioritisation) [17], have dubious relevance in their settings [17–19], or that failings in the conduct or reporting of trials mean that they do not provide the reliable and robust evidence that they need. Some trials may simply be unnecessary [20]. This all represents an unacceptably wasteful approach to designing, running, analysing, and reporting trials. The problem of inefficiency in medical research is not new: Schwartz and Lellouch urged trialists to change the way they designed trials as long ago as 1967 [21], Altman pointed to the scandal of poor medical research in 1994 [22], and, in 2009 [23], Chalmers and Glasziou estimated that more than 85 % of the resources invested in medical research was being avoidably wasted. What has been lacking is a coordinated attempt to tackle inefficiency in clinical trials.

Main text

Trial Forge

Trial Forge (www.trialforge.org) aims to address the lack of an accessible evidence base around trial efficiency and quality. A one-day workshop, funded by the Network of MRC Hubs for Trials Methodology Research and the Health Services Research Unit at the University of Aberdeen, UK, was held in Edinburgh on 10 July 2014 to discuss the initiative. The grant holders of the MRC Hub funding (Marion Campbell, Mike Clarke, Athene Lane, Trudie Lang, John Norrie, Shaun Treweek, and Paula Williamson) invited 38 participants with experience in methodology and trial design, trial management, statistics, data management, clinical care, commissioning and publishing research, public and patient involvement, and providing trial support through trials units to the worship.

The aims of the workshop were as follows:

  1. 1.

    To share knowledge on resources that already exist with regard to efficient trials.

  2. 2.

    To share knowledge on guidance relating to trial design, conduct, analysis, and reporting.

  3. 3.

    To agree on the key processes of the trial pathway, that is, the major processes in the life of a trial.

  4. 4.

    To begin to suggest features that Trial Forge must have.

  5. 5.

    To promote awareness of Trial Forge.

  6. 6.

    To produce a statement paper on the Trial Forge initiative

As the workshop members were professional trialists, trial managers, statisticians, and others involved in trial design, conduct, analysis, and reporting and the discussions were of current practice, no formal ethics approval, or consent was deemed necessary.

How will Trial Forge work?

Discussion at the workshop highlighted several substantial problems, some of which are listed in Table 1. Trial Forge aims to remove or reduce these problems and others through targeted collaborative work. Some of the ways it will do this are listed in Table 1. Trial Forge will use a five-step process to identify and address gaps in knowledge about trial method:

Table 1 Trial Forge Examples of trial challenges and how Trial Forge could help
Full size table
  1. 1.

    Identify trial processes

  2. 2.

    Collate what is known about these processes.

  3. 3.

    Strengthen the evidence base by creating a methodology research agenda.

  4. 4.

    Collaborate to work through the methodology research agenda.

  5. 5.

    Dissemination.

Step 1 - Identify trial processes

Step 1 will identify the processes that make up a trial, starting with the main processes (for example, recruitment) and then breaking these down into smaller processes (for example, how to set the eligibility criteria for a trial, selecting the components of the recruitment strategy, identifying potential participants, and targeting appropriate recruitment strategies for them). This is similar to the process improvement approach taken by the British cycling team in its preparation for the 2012 London Olympic Games. Dave Brailsford, British Cycling's Performance Director at the time said when asked about the team’s approach:

‘The whole principle came from the idea that if you broke down everything you could think of that goes into riding a bike, and then improved it by 1 %, you will get a significant increase when you put them all together.’ [24]

There are very many processes involved in a trial, and learning about, and improving each of them may have a minimal effect on its own, but taken together, these improvements could have a much more profound impact.

Participants at the Edinburgh workshop produced an initial list of headline trial processes (Fig. 1) for which collating (and creating) research evidence would be beneficial. This list will form the starting point for Trial Forge work.

Fig. 1
figure 1

Key processes of the trial pathway (many of which are overlapping and non-linear). Suggestions from a one-day workshop held in Edinburgh on 10 July 2014. The placement and length of the bars gives an indication of when in the trial they start and end, though this is likely to vary greatly between trials

Full size image

Step 2 - Collate what is known about these processes

In Step 2, Trial Forge will either identify existing initiatives to collate what is known about individual processes or work to collate the evidence (which may include providing links to ongoing studies) and integrate reviews (and other relevant literature) using both quantitative and qualitative synthesis approaches [25–28]. For example, for help in choosing trial outcomes, Trial Forge would direct people towards the COMET (Core Outcome Measures in Effectiveness Trials [29], http://www.comet-initiative.org) Initiative. COMET has systematically reviewed published standardised core outcome sets for trials [30], and combined these in the COMET database with information on core outcome sets currently being developed. As another example, for help with choosing evidence-based recruitment interventions, the MRC Network of Hubs for Trials Methodology Research is funding a project to develop a searchable database containing published and ongoing research into recruitment. On a smaller scale, Cochrane Methodology Reviews, and other systematic reviews have brought together existing research in specific topic areas. These will be highlighted in Step 2. Epistemonikos (http://www.epistemonikos.org/en/), a website that links together systematic reviews, overviews of reviews, and primary studies to support health-policy decisions, is another example of how research evidence can be collated.

More generally, the Evidence-Based Research Network (http://www.ebrnetwork.org) is an example of an initiative that aims to promote the efficient use of existing research, especially through the use of systematic reviews [31] and information about ongoing research. Proposals for new research should be informed by systematic reviews, and reports of new research should be set in the context of updated systematic reviews.

Trial Forge will aim to apply quality criteria when pointing to external resources and when collating individual studies. How to do this will form part of the initial work of Trial Forge, though it is likely that GRADE [32] (a system for grading the quality of evidence and the strength of recommendations, particularly for guidelines) will contribute importantly. Different approaches to presenting evidence will be explored using methods developed by the GRADE Working Group where appropriate, and the methods used to present the information will be informed by work done with, among others, the Cochrane Plain Language Summaries [33], the GRADE Summary of Findings tables [34, 35], and the DECIDE project (a project that aims to improve the way research information is presented in guidelines, http://www.decide-collaboration.eu). This presentation work will also be evaluated.

Step 3 - Strengthen the evidence base by creating a methodology research agenda

Step 3 will focus on strengthening the evidence base by providing a platform to highlight key areas of uncertainty, which would enable individuals and research groups to suggest ways in which the uncertainties could be addressed. For example, we know less about the effect of recruitment interventions aimed at recruiters than we do about those aimed at potential participants [7]. Recruiters play a hugely influential role and can have a substantial impact on recruitment [36, 37], but there remains uncertainty about how best to address the issues and concerns that recruiters face [36–45]. One way to help fill this gap (and others) may be through the availability of standard outlines for Studies Within A Trial (SWATs). The design of SWAT-1 is for site visits by the principal investigator to increase or maintain recruitment [46].

Publishing protocols for methodology research, which can then be embedded in other studies, makes it easier for research groups to become involved in filling evidence gaps. Much of the intellectual work around the appropriate methodology research already will have been done by the authors of the protocol. A database of outlines for SWATs is being developed to improve access to these ideas [47]. Step 3 of Trial Forge will produce SWATs as well as link people to initiatives such as the MRC-funded Systematic Techniques for Assisting Recruitment to Trials (START) programme (http://www.population-health.manchester.ac.uk/mrcstart/), which is developing a platform to evaluate recruitment interventions simultaneously across many trials.

Finally, where evidence does not yet exist, information about these gaps will be systematically directed to funding agencies for consideration in their prioritisation processes. In the meantime, Trial Forge will provide a repository for experience and knowledge from the community of trialists, trial managers, and others about interventions and approaches that they believed worked well in their settings. Trial Forge will thus provide support for electronically linked communities of practice (for example, through question and answer and discussion sections on its website) to facilitate sharing of knowledge and experiences, especially when rigorous evidence to inform decisions is lacking.

Step 4 - Collaborate to work through the methodology research agenda

A methodology research agenda will have been created in Step 3. Step 4 will encourage wide collaboration among methodologists, trialists, and other relevant stakeholders to tackle this research agenda. For some processes in the trial pathway (Fig. 1), the agenda will be substantial and very challenging. A single research group or trials unit is unlikely to have the skills, capacity, or interest to take on a whole agenda. By bringing research groups together around a shared agenda, Trial Forge will minimise unnecessary duplication, focus work on topics shown to be most in need of attention (with a recent survey of the priorities of UK Clinical Trials Unit Directors providing a good starting point [48]), and identify groups with the necessary expertise to do the work. For example, groups could work together to evaluate an intervention described in a SWAT. This collaboration between groups may happen naturally through direct contact but could be facilitated by Trial Forge, for example by having a coordinator identify potential links and encouraging collaboration.

Step 5 - Dissemination

The value of the expanded evidence base will be realised in Step 5: when Trial Forge has identified or generated an important result from, for example, an up-to-date systematic review of relevant methodology research, people who need to know about it should be informed efficiently. For example, if, as a result of including new trial data to the Cochrane review of interventions to improve retention in trials [8] meant that there was now clear evidence that a particular intervention was effective, Trial Forge would help to ensure that this information is disseminated efficiently to trialists. A variety of dissemination routes will be used, for example, electronic mailing lists, a Twitter feed, presentations at the UK Clinical Trials Units Directors’ meetings and training courses. Dissemination routes are likely to need to change over time and may well need to differ depending on the trial process being addressed. An underlying principle will be that simply publishing the findings in a journal article is unlikely to be sufficient to promote uptake. To maximise the impact of this methodology research, Trial Forge will use evidence from implementation research to promote clinical and professional behaviour change interventions [49]. This step of Trial Forge will also be evaluated.

The five steps in Trial Forge will be iterative, especially since many trial processes are linked and because suggestions for change in one area may have consequences for others. Trial Forge’s own processes will also be evaluated and modified over time as we and others learn from our experience of using the five steps to reduce gaps in knowledge about how best to design, conduct, analyse, and report trials. Once started, Trial Forge should produce a steady stream of methodology innovations that address trial process problems of recognised significance to people involved in trials. Importantly, work, and prestige will not be concentrated in one place or group but will be distributed across a collaborative network. Groups engaging with Trial Forge will be encouraged to build up their own portfolios of methodology work in areas that match their interests and expertise.

Conclusion

Trial Forge aims to support active and regular engagement with people who design, conduct, analyse, and report trials in the UK and elsewhere. It will promote meaningful improvements in trial efficiency and greater potential for trials to improve health. Moreover, Trial Forge will support an informal network of interested trialists, who will meet virtually and occasionally in person to build capacity and knowledge in efficient trials design and conduct. It will aim to be the ‘go to’ website for summaries of what is known about trial methods research but also for a linked programme of applied methodology research that encourages people to collaborate to fill gaps in evidence.

Not all problems in trials need more methodology research. However, many aspects of trial design, conduct, analysis, and reporting could be subjected to research to identify the relative effects of alternative approaches and whether these aspects are scientific, methodological, or administrative; they all have uncertainties that could be addressed by research leading to greater evidence-based approaches than is currently the case. We believe that Trial Forge will maximise the effectiveness and efficiency of trials, increase the chances that they will produce reliable and robust answers, and minimise waste. Trialists share many of the same problems; Trial Forge is about working together to solve them.