FormalPara Key Points for Decision Makers

Often, recruitment rates are lower than anticipated, requiring more time, more funding or both to deliver the target sample size.

The costs of overestimating recruitment rates can be up to over AUD$2 million (260% above budget) for incomplete trials.

Accurately assessing recruitment in the trial planning phase is critical to avoid costly overruns and incomplete trials.

1 Introduction

The scandal of waste in medical research is, regrettably, not a new phenomenon nor one that looks to be disappearing any time soon [1]. It is estimated that up to 85% of all health research is avoidable waste [2] and about 50% of registered clinical trials remain unpublished years after completion [3]. In clinical research, which is expected to be an almost AUD$70 billion industry by 2025 [4], waste includes trials that are discontinued, do not reach their target level of power, or require much more time or much larger funding to complete due to slower than expected recruitment.

The need to minimise waste and deliver statistical power efficiently is particularly pertinent for placebo-controlled trials in surgery. Reported rates of discontinuation in surgical trials range from 15 to 43%, with low rates of recruitment being the most commonly reported reason [5,6,7,8]. Trials involving a placebo arm are also known to be difficult to recruit [9]. Previous research into the use of placebo-controlled surgery trials [10], including our own systematic review of such trials in orthopaedic surgery [11], have highlighted the conditions under which placebo surgery-controlled trials are indicated; and the associated ethical issues and risk of bias [11,12,13,14]. What is less well understood is the economic implications of recruitment difficulties for placebo-controlled trials in surgery.

Economics provide tools for optimisation that can help to reduce such waste. Consistent with a standard economic problem, a trial takes inputs such as recruitment sites and potential participants and turns them into trial output in the form of statistical power to answer a research question, while facing constraints of funding and time. However, to date, research into the economics of trials is surprisingly sparse. Connelly provides an economic framework of a trial that looks at the trade-off between the number and size of recruitment sites [15]. Value of information (VOI) literature applies economics to help quantify the benefit of research in terms of the improved information on which to make decisions [16,17,18], while optimal design literature investigates efficient design from an operations research perspective [19,20,21].

This paper adds to the literature by developing a theoretical economic model of the costs of trials and parameterising this model with data for placebo surgery trials collated from a range of different sources including a systematic review reporting on placebo-controlled surgery trial completion and timely conclusion [22], and primary data collection. The model is then used to estimate the planned versus actual rates of recruitment and costs of three proxy trials according to whether they met their target sample size and their timeframe to completion. In doing so, the paper aims to help trialists and funders better understand the conditions under which placebo trials are more likely to deliver power in a cost-efficient manner.

2 Methodology

2.1 Theoretical Model

2.1.1 The Economic Production Problem

Trials face fixed and variable costs (see Box 1) [23,24,25,26], and a set of constraints and objectives consistent with the classic economic production problem [15]. When designing a trial and applying for funding, trialists typically face a cost minimisation problem, with the required sample size (based on a required level of power) forming the constraint. The costs, both fixed and variable, to deliver that sample size are then estimated and ideally minimised.Footnote 1 During trial recruitment, the optimisation problem switches from cost minimisation with a power constraint to power maximisation subject to budget and time constraints. As the trial has been funded for a given time period, this forms the budget and time constraints.

figure a

2.1.2 A Theoretical Economic Model of the Cost of Trials

The economic power maximisation problem can be defined algebraically to maximise the sample size recruited (n), subject to cost (c*) and time (t*) constraints:

$${\max} \, n, \text{subject to} \, c \le {c}^{*}, t \le {t}^{*}.$$
(1)

The total cost of a trial (c) is a function of the number of sites (s), the fixed cost of establishing each site (ps), the variable cost of recruiting (pn), and the sample size recruited (n):

$$c={p}_{\text{s}}s+ {p}_{\text{n}}n.$$
(2)

The duration of a trial (t) is a function of the target sample size (n*), the rate of recruitment per site (rn) and the number of sites (s):

$$t=\frac{{n}^{*}}{{r}_{\text{n}}s}.$$
(3)

The “iso-power” line relates costs to time for a given sample size (n*), and is found by substituting the time constraint into the budget constraint:

$$c= \frac{{{p}_{\text{s}}n}^{*}}{{r}_{\text{n}}t}+ {p}_{\text{n}}{n}^{*}= {n}^{*}\left(\frac{{p}_{s}}{{r}_{\text{n}}t}+ {p}_{\text{n}}\right).$$
(4)

This highlights the inverse relationship between cost and time as shown in Fig. 1A. A shorter trial duration necessitates an increased number of sites to deliver the required sample size and power. More sites result in a higher total cost due to the fixed costs associated with establishing each new site. Conversely, a longer trial duration allows a given sample size to be recruited from a smaller number of sites, and therefore a lower total cost. If a higher level of power is required, a larger sample size will be needed, moving the iso-power curve outward from the origin.

Fig. 1
figure 1

Economic model for cost of a trial showing iso-power lines for varying scenarios

Full size image

2.1.3 Incorporating a Model of Recruitment

In a simple two-stage model of recruitment, a cohort is screened for eligibility, with those who are eligible then asked if they consent to participate in the trial. Over a given time period, the total number of eligible patients (e) is a function of the total number screened (k), and the probability of eligibility (ep):

$$e=k {e}_{\text{p}}.$$
(5)

Similarly, the sample recruited is a function of the number of eligible patients and the probability of consent (cp):

$$n=e {c}_{\text{p}}.$$
(6)

The variable cost of recruiting (pn) will depend on the numbers screened (k) and asked to consent (e), and the variable cost of screening (pk) and consenting (pc):

$${p}_{\text{n}}=\frac{{p}_{\text{k}}k+ {p}_{\text{c}}e}{n}.$$
(7)

Substituting e and k provides the relationship between the variable cost of recruiting and the probabilities of eligibility and consent:

$${p}_{\text{n}}=\frac{{p}_{\text{k}}+ {p}_{\text{c}}{e}_{\text{p}}}{{c}_{\text{p}}{e}_{\text{p}}}.$$
(8)

This highlights that the variable cost of recruitment is inversely related to the probability of consent and the probability of eligibility.

The recruitment rate per site (rn) becomes a function of the number of patients who are screened (kn), and the probability of eligibility (ep) and the probability of consent (cp):

$${r}_{\text{n}}= {k}_{\text{n} }{c}_{\text{p}} {e}_{\text{p}}.$$
(9)

This provides an indication of the efficiency of recruitment. An alternative metric used in the literature is the recruitment index (RI), which is defined as the number of days required to recruit one patient [27]. Both measures provide equivalent information. For rates defined over a 30-day (monthly) time period, the relationship between the recruitment index and recruitment rate is given simply by:

$$\text{RI}= \frac{30}{{r}_{\text{n}}}.$$
(10)

Substituting pn and rn back into our cost and time trade-off function highlights the inverse relationship between total cost of a trial (c) and the probability of eligibility (ep) and the probability of consent (cp):

$$c= {n}^{*}\left(\frac{{p}_{\text{s}}}{{r}_{\text{n}}t}+ {p}_{\text{n}}\right),$$
(11)
$$c= {n}^{*}\left(\frac{{p}_{\text{s}}}{{k}_{\text{n}}{c}_{\text{p}}{e}_{\text{p}}t}+ \frac{{p}_{\text{k}}+ {p}_{\text{c}}{e}_{\text{p}}}{{c}_{\text{p}}{e}_{\text{p}}}\right),$$
$$c= \frac{{n}^{*}}{{c}_{\text{p}}{e}_{\text{p}}}\left(\frac{{p}_{\text{s}}}{{k}_{\text{n}}t}+ {p}_{\text{k}}+ {p}_{\text{c}}{e}_{\text{p}}\right).$$

2.1.4 The Corner Solution

It quickly becomes apparent that a corner solution often prevails in the design of many studies: the cost constraint directs that the number of sites be minimised; the time constraint determines the minimum number of sites required to ensure the trial meets the time constraint. Figure 1A provides a graphical representation of the model with a planned target power of 80% with time and budget constraints at T* and C*, respectively. At points along the 80% iso-power line above the budget constraint C*, the duration of the trial would fall within the time constraint, but the trial costs would exceed the budget constraint. At points along the iso-power line below C*, the trial costs would fall below the budget constraint, but the trial duration would exceed the time constraint T*. If either of the constraints are relaxed, a higher level of power can be obtained. For example, Fig. 1B highlights that a 90% power could be obtained if the trial duration was increased to T**. In practice, recruitment rates may be slower than anticipated. This lowers the efficiency of translating time and funding into statistical power. For example, Fig. 1C highlights an example where the planned 80% iso-power line now represents a lower level of power (60%), and to achieve the 80% power requires increased time and/or budget. In the next section, we apply the model to three proxy placebo surgery-controlled trials to highlight how the model can be a useful tool for trial development.

2.2 Data and Applications

2.2.1 Application to Placebo Surgery Trials

We applied the model to placebo-controlled surgery trials to understand the cost implications of poor recruitment. Placebo-controlled trials in surgery face the same overarching power maximisation problem as non-placebo trials, but potentially face higher costs (see Box 2) [22, 28,29,30]. Time can also be a significant constraint, with a recent systematic review of placebo-controlled surgery trials finding three-quarters did not meet their target timeframe [22].

figure b

Three proxy trial scenarios were developed based on the outcomes (trial completion and timely conclusion) observed in the recent systematic review [22]:

  • Completed within timeframes

  • Completed with an extension to original timeframes

  • Incomplete

    For each of the above, the model was parameterised for both the target trial as it was planned, as well as the actual as-observed trial.

2.2.2 Data

There are a minimum of eight parameters required to specify the model (Table 1). The choice of which parameters to make exogenous (or fixed in the model) versus which parameters to make endogenous (i.e., calculated by the model) depends upon both the data available to the user, and the research question that is being posed. In this case, total trial costs were not known, and our research question was to determine how slower-than-anticipated recruitment impacted on total costs. We therefore endogenized total trial costs and let this be calculated within the model. Parameters that were exogenously specified were sample size, duration, and number of sites, based on median data from a recent systematic review of placebo-controlled surgery trials [22]; fixed and variable costs, derived from a bespoke cost survey of an Australian surgery trial [31] and reported in Australian dollars; and eligibility and screening probabilities, derived from relevant literature [32, 33]. Collectively, these parameters allowed us to calculate the remaining endogenous parameters, including the total cost of the trial, the number and rate of patients screened for eligibility, and the recruitment index, for the target trial as planned.

Table 1 Input data
Full size table

To model the actual trial rather than the target trial, we exogenized the actual rate of patients screened for eligibility at the target rate, and then used the model to determine what that the actual probability of consent was given the actual trial duration, sample size and number of sites. This strategy assumes that trialists more accurately predict the number of patients who will be screened than the number of patients who will consent.

Finally, we also used the model to estimate the marginal impact of a one percentage point improvement in the probability of consent on the costs and duration of placebo-controlled surgery trials and highlighted how this varies across the three proxy trials (completed, completed with extension, and incomplete).

3 Results

3.1 Modelling Analysis of Targeted and Actual Trial Parameters

  1. 1)

    Completed trials

    Figure 2 displays the results for a typical placebo-controlled surgery trial that was completed within its target timeframe. The trial required a sample size of 70 participants and estimated that a duration of 54 months and 10 sites would be required to achieve this sample. Using these parameters in the model, we calculated that the trial planned for a recruitment index (the time it takes to recruit one participant) of 231 days, and total costs of AUD$1.2 million (Table 2).

    However, while the trial was completed within target timeframes, 50% more sites were required to achieve the target sample size. We calculated that this increased the actual recruitment index by 50% from the anticipated 231 days to 347 days and increased the required funding to AUD$1.8 million (50% higher than anticipated). In this example, the budget constraint was the flexible constraint: faced with lower than anticipated recruitment, the trialists were able to secure extra funding to deliver the trial on time.

  2. 2)

    Trials completed but not within the target timeframes

    Figure 3 displays the results for a typical placebo-controlled surgery trial that was completed with an extension to the target timeframe. The trial required a sample size of 130 participants and estimated that a duration of 18 months and four sites would be required to achieve this sample. Using these parameters, we calculated that the trial expected a recruitment index of 17 days, and total costs of AUD$0.8 million (Table 2). However, to complete the trial, an extra site was added (25% increase), and the duration was doubled from 18 to 36 months. We calculated that this increased the actual recruitment index to 42 days (147% higher than the anticipated 17 days) and total costs to AUD$1.4 million (84% higher than anticipated). In this example, both the budget and time constraints were expanded in order to complete the trial.

  3. 3)

    Incomplete trials

    Figure 4 displays the results for a typical placebo-controlled surgery trial that was incomplete. The trial required a sample size of 200 participants and estimated that 48 months and three sites would be required to achieve this sample. Using these parameters, we calculated that the trial expected a recruitment index of 22 days, and total costs of AUD$0.9 million (Table 2). However, recruitment for the trial was substantially slower than anticipated. In an effort to complete the trial, an extra eight sites were added, which increased the total costs to AUD$3.1 million (267% higher than expected). After 48 months only half of the required sample had been recruited and the trial was stopped. We calculated that the actual recruitment index was 158 days, over 600% higher than the anticipated 22 days.

Fig. 2
figure 2

Target and actual iso-power curves for trial completed on time

Full size image
Table 2 Inferred total costs, recruitment index and marginal impact of consent
Full size table
Fig. 3
figure 3

Target and actual iso-power curves for trials completed after an extension

Full size image
Fig. 4
figure 4

Target and actual iso-power curves for incomplete trials

Full size image

3.2 The Impact of Recruitment on Costs and Duration

Figure 5 shows the relationship between total cost and the probability of consent. The partial differential is an inverse squared relationship: every one percentage point reduction in the probability of consent is more costly than the previous. Our three proxy trials highlight this clearly. At one end, we estimated the completed trial delivered a probability of consent of 27%. Here, a one percentage point improvement in consenting would have saved AUD$14,000. At the other end of the spectrum, we estimated the incomplete trial achieved a consenting probability of just 5%. For this trial, a one percentage point improvement in consenting would have saved over AUD$312,000.

Fig. 5
figure 5

Total cost versus probability of consent

Full size image

4 Discussion

4.1 Summary of Findings

We developed a novel economic model of the costs of trials and parameterised it with placebo-controlled surgery trial information obtained from a systematic review, a cost survey, and data from the literature. We compared target and actual trial performance for three proxy trials (completed, completed with an extension, and incomplete) based on the features of similar trials in the literature. We found that:

  • Successful trials more accurately anticipated the true recruitment rate that they achieved. Findings from Bunzli et al. [22] indicated that less than one in ten placebo-controlled surgery trials accurately anticipated their true recruitment rate, and those that got it most wrong were most likely to fail. Estimates from our model showed that incomplete trials misestimated their recruitment index by over 600% compared to 147% for trials that completed with an extension.

  • Successful trials were smaller, which suggested that problems in recruitment were multiplied when a large sample size is required. For example, completed trials aimed for and achieved an average sample of 70 participants while incomplete trials aimed for 200 participants but only managed to recruit half of that, even after increasing the number of sites.

  • The costs of overestimating recruitment rates were dramatic: all proxy trials had significantly higher costs than planned. It was estimated that additional funding of around AUD$600,000 was required for the completed and the completed with extension trials, and over AUD$2 million for the incomplete trials.

  • The relationship between recruitment rates and total trial costs was not linear, which indicated that every reduction in the probability of consent has an increasing impact on costs. Given that low rates of consent are common in placebo-controlled surgery trials, increases in the probability of consent can significantly reduce the costs of the trial.

4.2 Implications

4.2.1 The Economic Problem

Our model highlights the basic economic problem of turning inputs: time, funding, and sites, into consented trial participants, and ultimately statistical power to determine differences between trial arms. Unfortunately, in placebo-controlled surgery trials, it appears that this problem continues to confound trialists, and trials continue to be less efficient than planned. While some of the waste in clinical trials can be apportioned to increasing complexity and increasing prices [4], our analysis highlights the fundamental waste associated with inefficient recruitment. Potential methods for trialists and funders to reduce this waste are discussed below.

4.2.2 Setting Realistic Recruitment Expectations

Firstly, our results highlight the importance of accurately assessing recruitment in the trial planning phase as a necessary precursor for a successful trial and generation of robust clinical evidence. Getting this wrong can be costly, resulting in substantial sunk and opportunity costs to those involved in the trial as well as funders. Despite widespread recognition that recruitment is often the most challenging aspect of running a successful trial, many continue to set over-optimistic expectations in recruitment [34]. A review of discontinued trials reported over-estimation of eligible participants as the most common reason for recruitment failure [35]. This could have been mitigated at the trial planning phase through better recruitment prediction and/or prevented through a feasibility exercise to test the informed consent process [35]. Researchers need to make realistic estimates of the feasibility of recruiting patients for placebo surgery trials, the time it takes to recruit sites, and targeted number of patients. Although statistical models exist to help researchers predict patient recruitment at the design stage, it is unclear how commonly these predictive models are used as their complexity and the lack of informed data from existing trials may be barriers to their implementation [36,37,38]. Simpler calculations such as the recruitment index [27] can help guide the planning stages of a trial (particularly in budgeting and application process) and can be used as an indicator of feasibility.

Despite the growing amount of literature on strategies to improve recruitment rates, researchers lack opportunities to share and learn from the mistakes of others. These are often not published or reported in the clinical trial registries. Detailed reporting of recruitment strategies, actual and targeted recruitment rates, and difficulties encountered can provide helpful information to other researchers in designing and planning recruitment strategies. The cost consequence of an incomplete trial is high. It results in an obvious waste of funding, failure to provide a clear answer to the clinical question and breaks the implicit ethical contract between trialists and patients who participate for the potential benefits to society. Better strategies are required to improve the accuracy of recruitment prediction and trialists should be obligated to demonstrate the feasibility of achieving targeted patient numbers at the funding stage.

4.2.3 Funding Efficient Trials

Funders have a role to play in providing information to help trialists understand both what is realistic and what is efficient in terms of recruitment and trial design. Measures such as the recruitment index could be used to highlight minimum rates of recruitment that would be required in order to be considered for funding. Analysis of historical trials could be used to determine these thresholds. Large, publicly available databases that record both protocols and recruitment outcomes for historical trials, such as the ClinicalTrials.gov database, could be used to examine the value of such thresholds when determining the feasibility of planned trials [39,40,41].

Funders may also consider pilot funding. As it is common for clinical trials to run a smaller scale pilot study to support the development of a future definitive randomised controlled trial [42], it may be reasonable to require trials to provide pilot information that demonstrates the capacity to recruit. This could include surveying the targeted population to explore willingness to participate in the planned trial, involvement of patients in trial design and planning, testing of eligibility criteria, demonstrating a robust recruitment strategy (screening and consenting procedures) and assessment (along with agreement) from sites that can support the research [35, 43, 44]. It would also be helpful to estimate the key aspects of the research costs such as the variable cost of recruiting each patient. Pilot studies could not only demonstrate clinical feasibility, but also provide information on the likely level of resources to conduct the study. This could form the minimum criteria of assessment required in demonstrating the capacity to recruit (and likelihood of success) to which researchers, ethics review boards and funders can apply.

One area in which there has been increasing traction is the involvement of patients (consumers) in medical research. Patient involvement in clinical trials has the potential to improve rates of recruitment [45] and funding bodies appear to be taking steps toward this by encouraging the submission of plans for patient and public involvement to obtain funding.

As the incremental cost of the total trial cost increases substantially with lower recruitment rates, funders may consider part payment of funds, with continuation and release of remaining funds after demonstration of progression in the first year. The recruitment index can also be used to track recruitment efficiency over the course of the trial. Funders can potentially build such reporting criteria as part of their funding agreements. This can potentially minimise losses on trials that are likely to overrun (either or both cost and time) or be incomplete.

4.2.4 Using Economic Models

The economic model builds on the earlier theoretical work on Connelly [15] who applied an economic lens to trials. It provides a working framework to help guide researchers and funders alike to realise the impact of recruitment rate on budgetary and time constraints and their likelihood for discontinuation. It can be used to assist in building the trial budget and in providing indicative boundaries within which a trial should succeed. Capturing changes to total trial costs and duration as a result of changes to recruitment rates during the life of the trial can potentially indicate the point when it is no longer feasible to continue with the trial to avoid further sunk costs. Such information can be valuable to funders providing additional confidence in the success of the trial within the pre-defined scope, time, and budget as well as signals that a trial is likely to be completed in time and within budget. This economic model presented can clearly demonstrate the trade-off between cost and time in delivering statistical power. For instance, in environments such as COVID-related trials with the need to answer critical clinical questions within short time frames, more sites would be needed to reach the targeted sample size quickly and, in such instances, cost cannot be an absolute limiting factor.

4.3 Limitations

This study presents one of the few models of the economic problem of trials. In doing so, simplifications were made in the model, and a range of data parameterisations were assumed in the analysis. The results presented here are indicative only, presented to highlight the usefulness of the economic model, the orders of magnitude of impacts, and the importance of accurate assumptions in the trial planning stage. We note in particular that some of the cost parameters held constant in this analysis would in fact vary by trial, which would likely affect the estimates of total cost, and marginal impact of consenting presented here. Similarly, the probabilities of eligibility and consent will vary widely depending on the specific criteria of the trial, and who is invited for screening. Despite this, the application of the model is flexible depending on available data and the research question to be answered. Although the costs used in the model were from a (non-placebo) surgical trial, personal communications with experienced trialists and from shared experiences within our team (PCh, MMD, ZB, SL) indicated that the fixed and variable costs of placebo-controlled surgical trials were unlikely to vary substantially from non-placebo surgical trials. Finally, because this model focused on recruiting and the costs of trials, the benefits of trials were largely simplified. The analysis presented here contains no consideration of the expected value of the research, or VOI analysis. To date, applications of VOI analysis remain limited [46]; however, it is an important methodology to help improve the allocative efficiency of research funding. Future models of trials could potentially bring together both costs and benefits within one economic model to consider not only the costs of delay, but the lost opportunity that costs of delaying research and associated decisions have on the effectiveness of the intervention.

5 Conclusion

Waste in clinical trials continues to present problems to researchers and funders alike. Placebo surgery-controlled trials present a range of further challenges for trialists. The economic model presented here clearly shows the trade-offs between time and cost. More often than not, recruitment rates tend to be lower than anticipated. Either more time, more funding, or both are required to deliver the target sample size. Getting this wrong can be costly, resulting in substantial sunk and opportunity costs to those involved in the trial, to funders and society. More consideration needs to be given to the accurate planning, reviewing, and funding of these trials to avoid costly overruns and incomplete trials. Efforts to increase recruitment, including raising the probability of consenting for placebo surgery-controlled trials, have large potential benefits.