Background

Surgical research is fundamentally challenging because surgery is a complex intervention. Unlike in pharmacological studies, surgical interventions are more complicated to standardise, define and compare with other interventions [1]. The challenges of designing, conducting and evaluating surgical trials have been well documented [2,3,4,5,6,7] and may be considered largely related to methodological or cultural issues. Key methodological challenges of surgical trials include recruitment, intervention stability and standardisation, and outcome selection and/or measurement. Cultural issues often compound methodological challenges and include problems with equipoise amongst surgeons, inexperience recruiting to Randomised Controlled Trials (RCTs) and lack of understanding of the multidisciplinary (non-clinical) expertise needed for definitive trial success. Although many of the practical and methodological challenges of evaluating surgical interventions are common to the assessment of all complex, non-pharmacological interventions, surgery is unique in that multiple challenges often coincide [2].

Pilot and feasibility studies are increasingly recognised as a key component for the success of subsequent definitive trials (studies appropriately powered to achieve an effect size and consequently definitively answer a research question). Definitions of the terms ‘pilot’ and ‘feasibility’ have evolved in the literature over time, with terms often used interchangeably and without universally accepted definitions [8,9,10]. Several major funders in the UK such as the National Institute for Health and Care Research (NIHR) and the Medical Research Council (MRC), have now adopted the conceptual framework to classify pilot and feasibility studies published by Eldridge et al. in 2016 [8]. However, for the purposes of this work, pilot and feasibility studies (PAFS) are defined broadly as ‘Any research undertaken before a main study that is explicitly intended to inform the design and/or conduct of a future main study.’

PAFS have value in informing the design and conduct of surgical trials because they face unique complexities, and often interacting uncertainties surrounding the design, conduct and completion of trials, meaning there is an even greater need to consider if and how surgical trials can be improved. Indeed, funders encourage consideration of feasibility before an agreement to fund a definitive trial is reached [11]. PAFS may help avoid poor research design, conduct and analysis, all of which are known to contribute to significant research waste [12,13,14,15,16]. By avoiding common problems such as the inability to recruit and a corresponding reduction in statistical power, excessive attrition due to intolerable procedures and cross-over between treatment groups, it has been suggested that PAFS may reduce the proportion of failed trials [17].

Whilst published guidance from the MRC [1, 18] and the IDEAL (Idea, Development, Evaluation, Assessment, Long-term follow-up) collaboration [19, 20] emphasises the importance of PAFS for trials of complex interventions such as surgery, accessible practical guidance tailored specifically to optimally design and undertake PAFS for surgical trials is lacking. The conduct, reporting and publication of PAFS in surgery remain rare [21, 22], and guidance such as the IDEAL recommendations, a framework for evaluating and reporting surgical innovation, has yet to lead to a demonstrable improvement in the performance and publication of surgical PAFS [23]. Furthermore, there is no surgery-specific guidance endorsed by funding bodies or professional membership organisations such as the Royal College of Surgeons (RCS) tailored to surgeons participating in or contemplating designing PAFS.

Accessible guidance specifically for surgeons, which distils the important methodological messages for designing and conducting PAFS into a practical useable framework, is needed. The aim of this work was, to generate a detailed understanding of the challenges of PAFS in surgery and to develop clear and practical recommendations for surgeons for how to optimise the design and conduct of surgical PAFS in the future. Such guidance will improve the understanding of the true purpose of PAFS in the context of surgical research, drive up the quality of research applications, optimise reporting and ultimately improve the quality and value of surgical RCTs, thereby significantly reducing research waste.

Methods

Our recently published 10-year review of NIHR-funded surgical PAFS [24] established that the full potential of PAFS to address the uncertainties and challenges specific to undertaking surgical trials is yet to be achieved. Findings from this review and from in-depth qualitative interviews with key stakeholders exploring the challenges and barriers to undertaking PAFS in surgery were synthesised to identify key factors contributing to sub-optimal surgical PAFS and develop practical recommendations for the design and conduct of PAFS in surgery. The synthesis triangulated data from both the quantitative review and the qualitative interviews, with available guidance on the importance of PAFS for trials of complex interventions [1, 19, 20] and wider methodological literature on PAFS more generally [8, 25,26,27,28,29]. Figure 1 illustrates the study process.

Fig. 1
figure 1

Study process of data collection, synthesis and analysis

Full size image

Interview study participant sampling

A list of potential participants was created by considering a long list of trial team members of studies included in our published 10-year review of NIHR-funded surgical PAFS [24]; Senior members of clinical trial units (CTUs) and RCS Trials Centres in the UK; Senior authors of published literature on PAFS work methodology; Senior editors of surgical journals and those publishing PAFS specifically and; Senior funding panel members of major funding bodies in the UK including NIHR, Cancer Research UK (CRUK), Chief Scientist Office (CSO) and Arthritis Research UK (ARUK). Senior participants were deliberately targeted as it was hypothesised they would have the greatest breadth and range of experience in the area of PAFS design and conduct, to consequently allow the extraction of information about the challenges and barriers to actually conducting pre-trial research from those with the most extensive experience. Participants were purposively sampled to achieve maximum variation based on the geographical place of work, clinical vs non-clinical expertise; clinical speciality (if applicable), areas of expertise and research roles. Snowballing sampling was also used, by asking participants during the interviews, if they had suggestions for other participants who may have alternative, relevant or important perspectives for this work.

Interview data collection and analysis

Interviews were conducted either face-to-face or by telephone at times and locations convenient to the participants, using a topic guide. This semi-structured approach provided a tool to frame the interviews and offer prompts for the interviewer whilst also allowing key topics of importance to participants to emerge naturally and be further explored (Appendix 1). Data analysis used an inductive thematic approach underpinned by the principles of grounded theory [30] using NVivo 10 software [31]. Sampling, data collection and analysis were undertaken concurrently and iteratively until no new themes emerged and data saturation was achieved.

Results

Demographics of Interview study participants

A total of 33 participants were invited to participate in an interview of whom 28 (85%) expressed an interest and 27 (81%) consented and were interviewed. These included 18 (67%) males and 9 (33%) females. Of the interviewed participants, 11 (41%) were surgeons, 16 (59%) methodologists and 20 (74%) funders. Surgeon participants included representatives of a broad spectrum of surgical specialties from centres across the UK. All the surgeon participants were currently involved in surgical research, but experience of involvement in PAFS varied widely. Trial methodologist participants included CTU directors and trial statisticians, all of whom reported experience in designing and conducting trials of complex interventions and PAFS, with a smaller proportion having had specific experience in surgical trials (3/16).

More than two-thirds (20/27) of participants interviewed had current or recent experience of membership on a UK research funding body panel including as a panel chair (n = 6), deputy chair (n = 3) or panel member (n = 12). More than half (15/27) were members of journal editorial boards and most (24/27) also currently held a professorial position at a UK university. Table 1 shows further interview participant demographics.

Table 1 Interview participant (n = 27) summary of demographic characteristics and surgical research/trials experience
Full size table

As shown in Fig. 1 and described in the methods, phase 3 of this work synthesised the findings from our published 10-year review of NIHR-funded surgical PAFS [24], and the n = 27 qualitative interviews with key stakeholders, as well as available methodological guidance on PAFS [1, 8, 19, 20, 25,26,27,28,29]. This synthesis resulted in four root causes for why PAFS in surgery are  not currently optimised as shown in Fig. 2. These root causes emerged from the data as issues relating to (1) understanding the full scope of PAFS; (2) the design and conduct of PAFS; (3) the reporting of PAFS; and (4) the undervaluation of PAFS by surgeons, journal editors, academic institutions and sometimes funders. In addition to the root causes identified for why PAFS in surgery are not currently optimised, compounding factors were identified which are linked to both the root causes and to each other. These factors relate to the challenges of (1) current guidance and (2) cultural issues surrounding both surgical research in general and PAFS more specifically. These challenges can impact at different and multiple points in the cycle, and thereby act as barriers to improving research practice (see Fig. 2). Each of the root causes and compounding factors identified by this work are described in detail below, supported by participant quotes (see Table 2) and references to the other data sources synthesised to produce these results.

Fig. 2
figure 2

Cyclical model of sub-optimisation of PAFS illustrating the linked and co-influential root causes and compounding challenges underpinning why surgical PAFS are not currently optimised

Full size image
Table 2 Quotes from interview participants
Full size table

Root cause 1: surgical PAFS are not optimally understood

A key finding from all data sources was that the purpose and scope of PAFS were not well understood by the surgical research community. The analysis of NIHR-funded PAFS protocols [24], for example, demonstrated that nearly a quarter of PAFS studies planned to conduct formal hypothesis testing (8/35, 23%). Interview data confirmed that many surgeons perceived PAFS to be small underpowered RCTs, designed and reported with the opportunity to test certain outcomes including safety and effectiveness [see Table 2 Quotes 1 and 2]. Whilst methodologists understood the full scope of PAFS to explore the breadth of areas of uncertainty about a potential main trial [see Table 2 Quote 3], surgeons were generally less specific about the multiple areas of uncertainty that could be explored and tended to focus on recruitment as the primary area of uncertainty that could be addressed [see Table 2 Quote 4].

Root cause 2: surgical PAFS are not optimally conducted

Misunderstanding of what PAFS are and why they should be done impacts the range of areas of uncertainty that surgeons will seek to explore in PAFS. Consequently, PAFS in surgery are not optimally conducted [see Table 2 Quote 5]. In particular, there is a tendency for surgical PAFS [24] to focus on issues that are generic to all trials, such as recruitment, rather than exploring and addressing key uncertainties of specific relevance to surgical trials, such as intervention stability and the learning curve. Interviews with methodologists highlighted that surgeons applying for funding to undertake PAFS tended to follow ‘example’ lists provided by funders of what to consider investigating in PAFS, rather than thinking about the specific needs of their trial [see Table 2 Quote 6]. The importance of exploring the intervention protocol (which also includes careful consideration of co-interventions) in surgical PAFS was particularly underappreciated [see Table 1 Quote 7].

Root cause 3: surgical PAFS are not optimally reported

Both the quantitative and qualitative work provided evidence that surgical PAFS are currently not well reported. Data from the review of NIHR-funded surgical PAFS illustrated that PAFS in surgery is under-reported, with only two-thirds of surgical PAFS studies funded by the NIHR between 2005 and 2015 publishing study findings [24]. In addition, interview findings demonstrated that PAFS are still masqueraded as full RCTs when submitted to journals and that underpowered RCTs are badged as PAFS a posteriori [see Table 2 Quote 8].

Root cause 4: surgical PAFS are undervalued

The interview study provided an explanation for why PAFS might be sub-optimally conducted and reported [24], by illustrating that PAFS were undervalued by all key stakeholder groups. Funders, for example, perceived that many PAFS had historically been conducted as standalone pieces of work with no intention of the study team or funders to progress to a main trial. PAFS were consequently undervalued as being ineffectual and not worth investment [see Table 2 Quotes 9 and 10]. Similarly, academic institutions were perceived to undervalue PAFS, considering them low-impact studies, which do not contribute significantly to the Research Excellence Framework (REF) as high-impact papers, and often have no outputs at all [see Table 2 Quote 11]. This is perpetuated by journal editors and peer reviewers, some of whom undervalue PAFS and consider them of limited interest as they do not offer definitive practice-changing results [see Table 2 Quote 12]. Such editorial practice may perpetuate the cycle of misunderstanding; if definitive results are requested, authors may feel compelled to produce them to achieve publication, thus small underpowered RCTs veiled as PAFS will continue to litter the literature, which further perpetuates misunderstanding [see Table 2 Quote 13]. Whilst there is now this journal dedicated to the reporting and publication of PAFS (The Journal of Pilot and Feasibility Studies [32]), without investment from journal editors, academic institutions and funders to drive the importance of accurately publishing pre-trial work, the cycle of sub-optimisation of PAFS will continue.

Compounding factor 1: challenges with current guidance

The interviews identified challenges with both guidance provided by funders, and information in the methodological literature conceptualising the types and purposes of PAFS and describing methods for reporting PAFS [8, 25,26,27,28,29]. The funder guidance was perceived as being limited, variable and sometimes contradictory to the definitions given in the methodological guidance [see Table 2 Quote 14]. Whilst most methodologists recognised the extensive methodological work already undertaken [8, 25,26,27,28,29], many felt this work to be inaccessible and poorly disseminated to surgeons [see Table 2 Quote 15].

There was limited awareness of the existence of the current methodological work amongst surgeons. The few surgeons who did mention this perceived the methodological literature as largely theoretical, generic and difficult to operationalise, thereby making it mostly unhelpful [see Table 2 Quotes 16]. This finding indicates that the methodological work is poorly understood and not widely acknowledged beyond the methodological community [see Table 2 Quote 17]. Guidance from the IDEAL collaboration [19, 33], widely considered as the conceptual work most aligned with surgeons and surgical trials, was not perceived in the interview study to be widely accepted or utilised amongst surgeons [see Table 2 Quote 18]. It is encouraging that the newest NIHR guidance first published in 2019 [34] does now signpost and reference the underpinning methodological work in this area.

If available methodological guidance is not effectively operationalised to be of practical use to surgeons [see Table 2 Quote 17], its inaccessibility is compounded, further adding to confusion so that it is consequently misunderstood or ignored. In addition, surgeons recognised significant barriers to being able to access Clinical Trials Units (CTUs) and methodological expertise, which was often perceived as not possible within PAFS funding envelopes [see Table 2 Quotes 19 and 20].

Compounding factor 2: cultural challenges

Many of the existing challenges for surgical trials have already reported [2,3,4,5,6,7] impact on PAFS and, in doing so, make PAFS potentially even more relevant in surgery. There has been a blossoming culture of surgical research partnerships and cross-specialty collaboration in recent years through, for example, the formation of nationwide surgical trainee research collaboratives and the RCS of England Surgical Trials Initiative [35]. Such cultural changes have undoubtedly contributed significantly to raising the profile of surgical research and, more specifically collaboratively conducting surgical trials [36,37,38]. However, of all clinical research funded by the NIHR Health Technology Assessment (HTA) and Research for Patient Benefit (RfPB) programmes from 2005 to 2015, only 10.4% (140/1341) were studies where surgery was the main intervention [24]. Most funders felt that the reason surgical research was less frequently funded was because the questions being asked were not important enough to the National Health Service (NHS) or to patients [see Table 2 Quote 21]. In contrast, surgeons perceived a lack of surgical representation on funding panels, and competition with translational science and experienced research teams for funding, as significant barriers to fair funding opportunities [see Table 2 Quotes 22 and 23].

Surgeons perceived other barriers to funding related to the relative lack of regulation for the formal evaluation of new surgical procedures and surgical devices. This was observed to have led to a lack of research infrastructure within the industry, resulting in fewer avenues for funding surgical research when compared, for example, to pharmaceutical research [see Table 2 Quotes 24 and 25].

Most funders still offer ‘uncoupled’ funding, where a PAFS is funded without a firm promise of funding for a subsequent main trial. This system was perceived by most interview participants as inefficient, due to the additional time and resources needed to perform standalone pre-trial work, and therefore a further barrier to completing PAFS [see Table 2 Quote 26].

Finally, both surgeons and methodologists perceived undertaking PAFS to lengthen the process of trial research, meaning answers to important questions took longer to attain. Consequently, it was considered that the research question may become obsolete before pre-trial work is completed, particularly in fast-moving clinical areas such as surgery [see Table 2 Quote 27 and 28].

Discussion

This is the first published work to specifically consider current research practice for PAFS in surgery and to explore the explicit challenges and barriers preventing optimal conduct of PAFS in surgery. This work has identified four key areas for improvement to research practice necessary to optimise future PAFS in surgery. These have informed the development of broad recommendations, summarised in Table 3, which require a wider cultural shift in research practice amongst funders, academic institutions, regulatory bodies and journal editors, as well as amongst surgeons. Whilst the recommendations from this work are focused on PAFS in surgery, many may be relevant to the wider context of complex interventions as a whole. As part of a future consensus process, this question could perhaps be addressed, especially in light of similar work in other areas [39,40,41].

Table 3 Recommendations for change to improve wider research practice around P/FS
Full size table

Examining the literature, it seems PAFS may be less commonly done in countries outside the UK. A systematic review by the methodology group which produced the conceptual framework of the definitions of PAFS, looked at the quality of reporting of 18 pilot and feasibility cluster randomised trials conducted and published between 2011 and 2014 [42]. This study found that half (56%) were set in the UK, with all other countries represented only once, apart from Canada (three studies) and the United States of America (USA) (two studies). In addition, it was noted in our study, that the UK-produced methodological guidance [8, 26] was perceived to have not been incorporated into practice by authors from overseas yet.

Whilst the focus of this work was entirely on research and funding practice in the UK, it is perhaps reasonable to suggest that the UK is leading the way in developing a methodology for the design and conduct of PAFS and that with further exploration and collaboration, the findings of this research could well be relevant to researchers in other countries. A limitation of this work is that only senior surgeons and methodologists were sampled. Surgeons leading the trainee surgical research collaboratives and research-naïve surgeons for example may have provided differing perspectives and potential solutions to the challenges encountered. However, the well-documented issues with inappropriate reporting of both underpowered RCTs as PAFS and vice versa [43,44,45] indicate a widespread misunderstanding of the value and purpose of PAFS. It was therefore deemed important to concentrate on extracting data from the most experienced and data-rich sources, hence focusing on experienced participants for the interviews, and systematic analysis of NIHR-funded PAFS (as opposed to performing a traditional systematic or narrative review of the literature). In addition, whilst patients were not involved in this work as the aim was to specifically explore the methodological and cultural barriers and challenges of completing PAFS from a professional perspective, involving patients in the design and delivery of PAFS is vitally important, and future work in the area will need to include the patient perspective.

Conclusion

This work identified the need for accessible, operationalised guidance for surgeons designing and conducting surgical PAFS. Our ‘Top Tips’ guidance tool for surgeons (Fig. 3) offers a practical framework for surgeons designing and writing funding applications for PAFS. The guidance operationalises and bridges the current gap between the available methodological guidance and the broader recommendations for improving research practice made here (see Table 3). These top tips include defining the purpose of PAFS, identifying uncertainties of specific relevance to surgery to be considered and engaging with methodologist support early and systematic reporting of PAFS, with references to key methodological resources. Both the recommendations for cultural changes (Table 3) and the practical guidance tips for surgeons (Fig. 3) are intended to optimise future best research practice around the design and conduct of surgical PAFS. Adoption of these recommendations will, therefore, facilitate successful and efficient surgical trials in the future and, ultimately, improve the evidence base for surgeons and patients.

Fig. 3
figure 3

Top Tips for surgeons designing and conducting pilot and feasibility studies

Full size image