Background

Health behaviour interventions, such as those which promote behavioural nutrition, increased physical activity, safe sexual practices, and help or prevent individuals from using tobacco, alcohol or other substances have the opportunity to improve individual health behaviours and prevent related mortality and morbidity [1]. However, interventions must be successfully implemented in real world contexts and ‘scaled-up’ to achieve population health benefits. Scale-up is defined by the World Health Organisation (WHO) as “deliberate efforts to increase the impact of health service innovations successfully tested in pilot or experimental projects to benefit more people and to foster policy and program development on a lasting basis.” [2].

Many effective health behaviour interventions are never delivered in the real world [3,4,5], and few are delivered at a scale that may have the capacity to achieve population wide reductions in related health risk [4, 5]. Implementation and scale-up frameworks identify a range of factors thought to be important for successful scale-up [6,7,8,9,10]. These include the characteristics of the intervention, such as the scientific quality (or risk of bias) of its evaluation, evidence of its effectiveness, and intervention flexibility [6, 10]. Efforts to scale-up an intervention may also be facilitated when it is supported and informed by a program of formative evaluation, efficacy and effectiveness trials, dissemination research and costing prior to scale-up [10,11,12,13] however this is seldom available [14].

Factors related to the characteristics of the setting where an intervention is to be delivered [7], have also been suggested to facilitate successful scale-up [6, 15, 16]. For example, settings with more formalised leadership and infrastructure (e.g. training, delivery systems, technical resources like those found in schools) may aid intervention implementation and scale-up [6, 17], The involvement of end-users across the phases of research is also recommended [18,19,20]. This improves the compatibility of the intervention to the context or setting where it is to be implemented, and the collection and reporting of information relevant to end-user decision making [21].

Strategies to disseminate the findings of intervention research may improve the likelihood that such interventions are discovered, and implemented, including at scale. International surveys report researchers often use seminars or workshops, face-to-face meetings, media interviews and targeted mailings to disseminate the findings of their research [21, 22]. While research shows that comprehensive dissemination strategies directed at patients can improve the use of evidence to support patient health related behaviours [23], There is limited research examining the association or impact of such intervention trial dissemination strategies on the likelihood of subsequent implementation and scale-up of prevention interventions [22, 24].

In this context, the broad objective of this study was to quantify the frequency of implementation and scale-up of health behaviour interventions and the extent to which factors suggested to facilitate implementation and scale-up are associated with it [14]. Therefore, the objectives of our study are:

  1. A)

    To describe how often tested public health interventions are reportedly implemented in practice;

  2. B)

    To describe adaptations to interventions that are thought to facilitate implementation;

  3. C)

    To describe the frequency to which selected interventions are scaled-up; and

  4. D)

    To examine the association between trial characteristics (effectiveness, trial quality, settings, involvement of end-users, and dissemination strategies) and scale-up.

Methods

Study design

We administered a single cross-sectional survey to trial authors (May 2018-June 2019) from 41 countries who (co)authored published manuscripts reporting the effect of public health primary prevention trials [25]. To identify authors of trials, we searched the Cochrane Database of Systematic Reviews for reviews of public health primary prevention interventions. Cochrane reviews were used as the sampling frame to provide international representation and broader coverage of health research and undertake comprehensive and systematic methods to identify all relevant studies. Cochrane reviews were eligible for inclusion if they: targeted nutrition, physical activity, sexual health, tobacco use, alcohol or other substance use; were set in any organisation; and were published between 2007 and 2017.

From the 42 relevant Cochrane reviews (see Additional file 1 for a full list of Cochrane reviews) we extracted trial and author details of interventions meeting study eligibility criteria. We included and extracted author information from trials that met our eligibility criteria, that is trials with a parallel controlled design (randomised or non-randomised) that: had some or all components of the intervention delivered in a setting (e.g. hospital, school, or workplace setting); examined the effects of a preventive health intervention (i.e. those targeting nutrition, physical activity, sexual health, tobacco use, alcohol or substance use); and was published between 2007 and 2017 (to allow time for implementation and scale-up to occur). Trials which did not meet this criteria were excluded.

Recruitment and data collection

We invited via email, the corresponding, first, second or senior authors from each eligible trial to participate in the study. Authors were invited to complete a Computer Assisted Telephone Interview (CATI) survey or an online survey using an emailed link. All data were entered using REDCap [26], a web survey hosting service. We sent respondents two telephone and/or email reminders (four and eight weeks after the initial contact) with a link to the survey, to maximise study participation rates [27]. Authors could nominate co-authors to complete the survey on their behalf. After four weeks, if we received no response from corresponding authors, we invited the first, second and/or last author of the trial manuscript (if different from the corresponding author) to participate.

Data collection and measures

The investigator team developed a study specific survey to assess implementation, adaptations, scale-up, and dissemination strategies used by those who delivered a primary prevention intervention following an iterative process. Survey items were grounded in the Payback [28] and Knowledge-To-Action (KTA) Frameworks [29], and similar studies in the published literature [30,31,32,33,34,35,36,37,38,39,40]. Data extraction was conducted by two review authors (SG and KM). Prior to completing data extraction, the data extraction tool was tested on a sample of studies until data extraction was harmonised.

Aim 1: describe how often public health intervention are implemented into practice. To describe how often tested public health interventions are reportedly implemented respondents were asked: “When the original trial finished, was the intervention implemented in other settings or organisations not involved in the original trial?” If respondents answered ‘yes’ they were asked to report; “How long from trial completion did it take for the intervention to be implemented into practice”.

Aim 2: describe adaptations to interventions that are thought to facilitate implementation. To describe adaptations to interventions that are thought to facilitate implementation, respondents who reported the intervention they trialled had been implemented in other sites, were asked: “Do you know if the intervention was adapted before being implemented within the < < insert organisation type > > other than those who consented to be involved in the original trial?” ‘Original trial’ in this case referred to any organisation that participated in the original study as a control or intervention site. Respondents who answered ‘yes’ then completed a series of items (see Additional file 2) that assessed adaptations as described by the Adaptome framework.

Aim 3: describe the frequency to which selected interventions are scaled-up. To describe the frequency to which selected interventions were scaled-up, participants were asked whether they received “Investment in the large scale delivery of the intervention by a government or non-government organisation?” This question captures the core dimensions of scale-up namely, it occurring on a perceived ‘large’ scale, and external investment and ownership [23, 42]. Trials of authors that indicated ‘yes’ to this item were defined as having been scaled up.

Aim 4: examine the association between trial characteristics and scale-up. To examine the association between trial characteristics and scale-up, we extracted information from published trial reports regarding the effectiveness of the intervention and the intervention setting, and extracted risk of bias (RoB) assessments published for each trial in the Cochrane review in which they were included. Items were included in the survey of corresponding trial authors to assess the involvement of end-users, and dissemination strategies employed. Specifically, we extracted data that described the effect of the intervention on primary and secondary trial outcomes using the following outcome hierarchy [43]. P-values were chosen as a measure of intervention effectiveness as they are a metric used to assess trial ‘effectiveness’ reported in the literature [44]. For trials where the primary outcome was not specified in the manuscript, we determined the outcome measure authors used to calculate sample size was the primary outcome. If sample size was not calculated and reported, we adopted the outcome described or inferred from the trial aim as the primary outcome. We selected adjusted p-values (if reported) over unadjusted p-values. We considered an intervention ‘effective’ if the trial reported a significant effect (p < 0.05), in the hypothesised direction, on a primary trial outcome. We considered analyses on secondary outcomes as hypothesis generating [45]. Trials were ‘potentially beneficial’ if change in the primary trial outcome was not significant, but change in one or more secondary outcomes were significant [45]. In instances where primary and sub-group analyses were reported, we included data only from the primary analysis.

We extracted RoB data for each trial from the Cochrane review where the trial was sourced (see Additional file 1 for a list of the Cochrane reviews). Cochrane RoB tools requires reviewers to assign high, unclear, or low for each domain of the tool. We categorised a trials overall RoB based on the most frequently applied classification across the domains. That is, trials in which RoB domains were most frequently categories as ‘high’ RoB were categorised as such. Where two or more RoB assessments were reported with the same frequency, the overall classification given to the trial was based on the higher RoB.

We extracted information about the setting in which the intervention was delivered (or where most intervention components were delivered) and classified these as education (schools and childcare settings), community, medical/healthcare, worksites, or other settings.

End-user involvement in the research was assessed by asking participating authors “To what extent were end-users involved in designing, conducting and evaluating the trial?” Respondent answers were scored as: 0 = ‘not at all’, 1 = ‘a little’, and 2 = ‘substantially’. Participants were asked if any of the strategies listed in Box 1 were used to disseminate findings from their trial [28, 30,31,32,33,34,35,36,37,38,39,40]. We counted the number of dissemination strategies used and used the total in our analyses. This made the variable linear, treating all dissemination strategies as having an equal effect.

Box 1. List of dissemination strategies participants (i.e. trial authors) were asked about at trial completion

• Plain language or lay summary

• Targeted presentations to end-users

• Knowledge broker used to communicate findings to end-users

• Education workshops conducted with end-users

• Education materials on how to use the study findings

• Media releases

• Results posted on institutional or study website

• Results posted on social media platforms (e.g. Facebook, Twitter, blogs)

• Publication of results in peer reviewed journals

• Research reports

• Presented at academic conferences, workshops or forums

Analysis

Data were collected and managed using REDCap electronic data capture tools, hosted at Hunter Medical Research Institute [26]. All analyses were undertaken with SAS v 9.3 [46]. We present descriptive statistics as numbers and percentages for categorical variables and means (standard deviation; SD) or median (quartile 1, quartile 3) for continuous variables, depending on distribution of the data. We calculated time to implementation in months using data provided in the open-ended question, often reported in months or years. Where a range of values was provided, we used the mid-point value for our analyses.

We used logistic regression to examine associations among trial characteristics and scale-up. The association of all variables with the scale-up was assessed in a (univariate) logistic regression model. All variables were then entered all at once into one multivariable regression model; this allowed the model to adjust for each independent variables estimated association with the scale-up. We report unadjusted and adjusted odds ratios and 95% confidence intervals (CIs) for each trial characteristic. We report p-values from the multivariable model.

Results

Searching the Cochrane database for eligible review resulted in 1154 review titles which needed screening, 509 of which were full text screened. Of these, only 42 reviews were identified for inclusion, resulting in a total of 208 individual trial included in the study (Fig. 1).

Fig. 1
figure 1

Illustration of the flow and final selection of reviews and trials included in our study

Full size image

Authors from 208 trials were invited to participate in the study, and we received completed surveys for 104 trials (n = 104 authors, 50% completion rate). Most trials focussed on physical activity and/or nutrition (55%) and were conducted in the education setting (67%) in North America (34%). A description of the trials associated with authors who completed the survey is presented in Additional file 3: Table S1.

How often tested public health interventions are reportedly implemented in practice Following trial completion, 49% (n = 49/99, 5 missing) of interventions were subsequently implemented by an organisation not involved in the original trial (Table 1). Average time to implement the intervention by another organisation following trial completion was 19 months (median = 12; range 0–72 months; n = 42, 7 missing).

Table 1 Author self-reported adaptations to the intervention or implementation strategies reported at scale-up
Full size table

Adaptations to interventions that are thought to facilitate implementation Fifty-four per cent of implemented trials (n = 25/46, 3 missing, Table 1) adapted their original intervention prior to it being implemented by an organisation not involved in the original trial. The two most frequently reported adaptations were, (1) adding intervention components, and (2) adapting the intervention to facilitate fit within the local service setting (48%; Table 1). Respondents indicated adaptations would positively impact intervention effectiveness (increase effectiveness; 32%) or diminish effectiveness (dilute effectiveness, 32%).

The frequency of which selected interventions are scaled-up One hundred authors responded to the question about scale-up (4 missing). Thirty-three respondents (i.e. 33%) indicated that their intervention was scaled-up—that is, there was external investment in the large scale delivery of the intervention by a government or non-government organisation.

The association between trial characteristics (effectiveness, trial quality, settings, involvement of end-users, and dissemination strategies) and scale-up Results from the unadjusted logistic regression indicate that the number of dissemination strategies increased the odds of scale-up by 1.48 with each additional strategy used. After adjusting for all other independent variables, the number of dissemination strategies remained significantly associated with scale-up (Table 2), with the estimate increasing to 1.50 for each additional dissemination strategy used. Scale-up was not significantly associated with other trial characteristics.

Table 2 Results of the logistic regression examining trial characteristics associations to scale-up (n = 49)
Full size table

Discussion

We extend a relatively sparse evidence base by quantitatively characterising implementation and scale-up of public health interventions, and by identifying factors associated with scale-up. We found that implementation and scale-up reportedly occurred in less than half of the trials included in this study, that the majority of interventions that were implemented in practice had been adapted, and that the number of dissemination strategies, but not the effectiveness of the intervention or study quality (as assessed via risk of bias) was associated with scale-up. Our contribution is an important one, given the substantial investment by granting agencies and governments worldwide in “one-off” pilot studies that do not yield community health improvements [47, 48].

We found that one third of published interventions in this study were scaled-up. This is a greater proportion than that previously estimated by Reis et al. that suggested 3% of physical activity interventions reported in studies were scaled-up [49]. We consider a few possible reasons for the greater likelihood of scale-up reported in our study. First, the sampling frame in our study were trials included in a Cochrane review, and were typically RCTs. Reis included a broader range of trial designs. Cochrane is a trusted and commonly used source of public health evidence [50], something made especially clear during the COVID-19 epidemic where their review evidence was sought by the World Health Organization, National Institute for Health Research and the Brazilian Ministry of Health [51]. Given this, trials included in such reviews may be more likely to be identified and their interventions adopted by health agencies. Second, there is no clear, threshold for determining when scale-up has occured [52]. It may be the moment an intervention is institutionalised at scale by being replicated in a different geographical area [49], or perhaps when it has reached large scale investment [49, 53, 54]. Therefore, perceptions of what constituted scale-up may have differed among respondents between ours and the Reis study. Studies may have used different criteria, tools and approaches to ‘count’ scale-up. For example, data that comprised the 3% of studies scaled-up was specific to physical activity interventions that had published outcomes of the effects of scale-up [49]. We included all public health related studies (only 21% were related to physical activity) and authors self-reported if their intervention was subsequently scaled-up (i.e. no published outcomes were necessary). When compared to a more recent review examining scaled-up nutrition settings-based interventions [5], the proportion of included studies that were scaled-up remains low when compared to other systematic reviews on the same subject (i.e. 10 included scaled-up nutrition studies [5] compared to 69 nutrition studies based in school settings in recent systematic reviews [55]). Nonetheless, the reported prevalence of scale-up in this study is encouraging, and suggests higher rates of this form of translation.

Adaptation for implementation and scale-up is often reported [4, 5, 41, 56]. In this study, adaptations to both function (‘core components’; actions considered essential to drive outcomes) [41] and form (e.g. mode of delivery; delivery preferences or actions considered not essential to outcomes) [57] were common. In public health research to describe core components of an intervention is rare [58], likely due to the absence of large trials that differentiate the specific contribution of intervention components to outcomes. The nature of adaptations to interventions or implementation strategies are also seldom reported, limiting our ability to compare across studies [59]. We recommend that consistent reporting of adaptations at scale-up of perceived core versus non-core components become standard practice. New literature is developing guidelines and frameworks to assist with this issue [60, 61].

We found that a greater number of dissemination strategies was associated with scale-up. Adherence to dissemination guidance such as the Interactive Systems Framework for Dissemination and Implementation [62] may further enhance the impact of investments in dissemination through ensuring the timing, source, message, and channel used in the dissemination strategies are appropriate to decision makers [63]. Future research investigating the effectiveness of specific dissemination strategies, individually and in combination, may help to improve the efficiency and impact of these approaches, with a recent scoping review highlighting there is a significant gap in the literature evaluating the impact of different strategies [22].

Interestingly, we found a lack of association between intervention effectiveness and scale-up when assessed against the significance of the primary trial outcome. This is concerning given that scarce preventive care resources comprise a miniscule proportion of the health care budget (1.9%, 5.8%, 3.0% and 5.2% in Australia, Canada, USA and UK, in 2017 respectively) [64]. The intervention may have reported beneficial effects on other outcomes considered important to end-users investing in their implementation and scale-up. Previous research has identified that a range of factors beyond effectiveness influence scale-up decisions including individual values, politics and finance [7]. Government officials report taking a ‘package proposal’ to politicians to scale-up health programs, and consider perceived end-user or community acceptability, feasibility and cost as well as effectiveness when making decisions [7]. Greater collection and reporting of such evidence may assist in facilitating program scale-up. Surprisingly, reported involvement of end-users was not associated with scale-up. Stakeholder engagement is thought to be essential to ensure the relevance of an intervention and its implementation success [65] and ‘beginning with the end in mind’ [66] seems a logical approach to ensure the research is warranted, has real-world applicability and the possibility that it will be sustained, in future. Greater exploration of this finding is warranted.

We acknowledge a number of limitations need to be considered when interpreting the findings of this study. First, scale-up is a long-term process and more recent trials (published up until 2017) included in this study may not have had sufficient time for this to occur. However, the range in the average time to implementation identified in this study (0–72 months) suggests instances of implementation that may occur, were likely to have been captured in this study. Nonetheless, more comprehensive studies that map the time to implementation and scale-up of public health interventions would help inform periods of latency for these outcomes and the appropriateness of data collection efforts to capture them. Second, we used Cochrane systematic reviews as a pragmatic approach to efficiently identify trials included in our study. However, such reviews often included randomised trial only and may include other criteria such as minimum periods of follow-up, or objective measures of outcome. As such, the findings of the review may not generalise to interventions and trials that may not meet such thresholds. A systematic review of primary studies with more open inclusion criteria would enable a more comprehensive and representative assessment of the objectives of this study. Finally, the impacts of dissemination strategies was assessed using a score based measure by summing the number of strategies employed. Analysis of the association of the individual dissemination strategies may have provided more informative information for the design of more potent approaches to dissemination. Future research should undertake this work.

Conclusions

It is essential to scale-up effective public health and health promotion interventions to maintain and preserve health at the population level. This study reports that both implementation and scale-up occurred in half of the interventions included from the sample, and that adaptation to these are common in the implementation and scale-up process. The frequency of adaptations highlight the potential that the effects of interventions may also be modified, and the importance of strategies to mitigate the risk of adaptations resulting in a ‘voltage drop’ [67]. While the study supports the use of comprehensive strategies to disseminate the findings of research trials to facilitate research translation, end-users must ensure that interventions selected for scale-up are indeed effective in order to achieve community health improvements.