Introduction

There is a significant variation among primary care providers in potentially harmful and relatively ineffective prescribing [1,2,3,4]. Prescribing of medicines such as antibiotics, opioids, anticholinergics, and non-steroidal anti-inflammatory drugs (NSAIDs) is associated with wide-ranging harms, including increased mortality, hospitalization, falls, dependence, and antimicrobial resistance [3, 5, 6]. It also represents poor value for money in resource-constrained healthcare systems [7].

A range of approaches has been used to address problematic prescribing, including financial incentives, electronic decision support tools, and educational outreach [1, 8, 9]. Audit and feedback is often used as a core component of these approaches; it aims to improve patient care through collecting and feeding back performance data to providers [10]. Feedback may improve prescribing in several ways, including alerting providers to the need for action, encouraging providers to think twice before initiating medicines, and triggering structured reviews to rationalize medicines for individual patients [1]. Audit and feedback offers the relative efficiencies of harnessing routine electronic health record data, often available for prescribing [11] and having a wide population reach. High-performing health systems tend to feature audit and feedback as an evidence-based, scalable, and relatively inexpensive strategy to encourage uptake of best practice [12].

Audit and feedback is generally modestly effective. A Cochrane review of 140 randomized trials found that audit and feedback produced a median of 4.3% absolute improvement in processes of care, such as prescribing [10]. However, there is a wide variation in effectiveness. A quarter of audit and feedback interventions had relatively large, positive effects of up to 16% while another quarter had no or even negative effects. Feedback effects are generally greater when the source is a supervisor or colleague, it is provided more than once, it is delivered in both verbal and written formats, and it includes both explicit targets for change and action plans. However, there are still many uncertainties about how to optimize audit and feedback [13]. Progress has been hindered by research waste in this field through a lack of cumulative learning from duplicative trials of audit and feedback [14]. This means that those responsible for designing and delivering quality improvement programs will continue to have unanswered questions about how to include audit and feedback for specific problems and circumstances [14].

To understand when audit and feedback is most likely to be effective and how to optimize it, we need to move away from two-arm trials of audit and feedback compared with control in favor of head-to-head trials of different ways of providing audit and feedback. Implementation laboratories use a learning health system approach, involving collaborations between healthcare organizations providing audit and feedback at scale, and researchers, to embed head-to-head trials into quality improvement programs. Implementation laboratories have been proposed to improve the impact of audit and feedback while simultaneously producing generalizable knowledge about how to optimize the effects of such quality improvement initiatives [15]. A “radical incrementalism” approach involves making sequential, small changes, supported by tightly focused evaluations to cumulatively improve outcomes. It is already used in public policy and in business. For example, Amazon and Google randomize potential customers to different presentations of their products online to understand what drives purchases. It is also highly applicable to healthcare [16]. While trials of different feedback methods have been embedded within existing audit programs [17], there is still limited experience in establishing audit and feedback implementation laboratories.

Our earlier work involved interviewing primary care staff (physicians, nurses, and managers) about a range of evidence-based recommendations and involved primary care physicians in developing an implementation package that included audit and feedback [18]. The intervention package successfully reduced high-risk prescribing in primary care [19]. Interviews with primary care physicians and prescribing data identified opioid prescribing as a high-priority topic for improvement [4, 20]. We adapted this feedback intervention and delivered a 1-year feedback campaign on opioid prescribing in primary care. Our controlled interrupted time-series analysis found that the number of adults prescribed any opioid (excluding those with coded cancer diagnoses) fell during the intervention year [Alderson SL, Farragher TM, Willis TA, Carder P, Johnson S, Foy R: The effects of an evidence and theory-informed feedback intervention on opioid prescribing for non-cancer pain in primary care: a controlled interrupted time series analysis. Submitted.]. We interviewed physicians and found our feedback campaign was well-received by practices and worked by alerting prescribers to the need for action, prompting prescribers to think twice before initiating opioids, and encouraging medication reviews of patients who may not benefit from opioids [Wood S, Foy R, Willis TA, Carder P, Johnson S, Alderson SL: General practice responses to opioid prescribing feedback: a qualitative process evaluation. Submitted.]. We also identified remediable weaknesses.

We are planning an audit and feedback implementation laboratory to address high-priority prescribing issues in primary care in the UK. Our overall program of work is strongly embedded in the primary care physicians’ perspectives and experiences. In this study, we sought wider service, research and public perspectives on prescribing priorities, and the feasibility and ethical challenges of establishing an implementation laboratory [21].

Methods

We used a two-stage Delphi consensus development process [22] to define key features of the planned implementation laboratory. The Delphi method is an iterative process which involves collecting, analyzing, and sharing the anonymous opinions of key stakeholders. It is useful when there is incomplete knowledge about a problem, particularly when the goals are to improve understanding and suggest solutions [22].

We identified potential participants as local primary care pharmacy leads responsible for medicine optimization in primary care, international audit and feedback researchers (affiliated to the Audit and Feedback Metalab [19]), and an existing primary care Patient and Public Involvement and Engagement (PPIE) panel with prior experience in implementation research [23]. This would ensure that our Delphi process would account for service, research, and public perspectives. We planned for 15–30 panelists in considering likely reliability and feasibility [22]. There is no agreement about the optimal size of a Delphi panel, with many including under 20 people [24, 25].

Using online surveys [26], we presented participants with questions and statements around priorities for an implementation laboratory and feasibility and ethical considerations (Additional file 1). These covered the following:

  • Current feedback practice—how feedback on prescribing is currently delivered to primary care practices (for medicine optimization leads only)

  • Priority topics for feedback—offering six options of antibiotic prescribing, opioids for chronic non-cancer pain, gabapentinoids, anticholinergic burden, prescribing safety indicators, and prescribing in low kidney function

  • Perceived usefulness of audit and feedback in improving prescribing and importance of improving its effectiveness

  • Perceived usefulness of different types of prescribing data, such as total numbers of prescriptions, numbers of patients prescribed medicines, and numbers of patients in high-risk subgroups prescribed medicines

  • Acceptability of different levels of randomization—at four levels of practices, practice networks (small groups of practices covering 50,000 patients), clinical commissioning group (larger area-wide groups of practices with responsibility for commissioning healthcare), and sustainability and transformation plan (44 larger geographical areas covering NHS England, now referred to as integrated care systems)

  • Acceptability and desirability of options for trial consent—including “opt-in” consent for primary care practices, “opt-out” consent for practices (so that they would be included in any trial by default unless they actively decline), consent at higher organizational levels (clinical commissioning group or sustainability and transformation plan), and waiving of consent

  • Acceptability and desirability of different methods of delivering feedback—such as paper-based hard copies, emailed PDFs, or online dashboards

  • Interest in finding out how effective different ways of delivering feedback would be—such as using different comparators, visual presentations, and additional education outreach and training

We drew on previous work describing potential challenges in setting up implementation laboratories and team brainstorming [13, 15, 21]. We piloted the statements with two feedback researchers and three medicine optimization professionals. For most statements, participants could rate according to one or more of importance (potential to improve patient safety and care), priority (based on whether there are existing interventions to improve safety and care), usefulness (to primary care providers), acceptability, and desirability (for research purposes). Statements were rated on a 1–9 scale, where “1” indicated the strongest disagreement and “9” the strongest agreement. We also provided an “unable to score” response if participants felt unable to rate a statement. Participants could also comment on the rationale for their decisions. We asked participants to complete the survey independently in round 1. Support for a statement was defined as 70% or more of responses scoring ≥ 7 and 15% or less responses rating the outcome ≤ 3.

We removed statements rejected as “not important” for round 2 to maximize efficiency. The remaining statements were then fed back to all participants via email. We showed each participant their own scores from round 1, as well as the median scores from all participants and for each of the three stakeholder groups [22]. Participants then independently re-rated each statement [27]. We analyzed the final responses for all participants and by each stakeholder group.

The University of Leeds School of Medicine Research Ethics Committee approved the study (ref no: MREC 18-049).

Results

Fourteen participants comprised eight females and six males, five medicine optimization leads, five feedback researchers (two from The Netherlands, two from Canada, and one from the USA), and four PPIE panelists out of 15, 52, and 10 invited, respectively (Table 1). All completed rounds 1 and 2, with a 100% completion rate (Fig. 1).

Table 1 Characteristics of participants
Full size table
Fig. 1
figure 1

Flowchart of the consensus process

Full size image

The medicine optimization leads reported considerable variability in the current prescribing feedback practice for primary care (Additional file 2). While all reported using a comparator, none reported using an online dashboard (linked or not to electronic health records), posting paper reports, nor initially providing intense feedback before reducing frequency.

Eighteen out of 55 statements were supported in round 1 (Additional file 3). All 55 statements were taken forward to round 2 as no statement reached the threshold for rejection. Thirty-eight statements were supported in round 2 (Table 2), while eleven had no clear support.

Table 2 Recommendations for a primary care prescribing implementation laboratory
Full size table

Prescribing priorities

All prescribing options, except gabapentinoids, were rated as important, and priorities for improvement, with antibiotics and opioids for chronic, non-cancer pain, rated as the highest priority (100% and 93%, respectively). All panelists agreed that feedback interventions were both useful (86%) and important (93%) as a method to improve primary care prescribing.

Ethical issues

The panel agreed including all primary care practices unless they actively withdrew from the implementation laboratory (opt-out) was both the most acceptable (100%) and desirable (86%) approach to consent. Individual practice consent (opt-in) was considered neither acceptable (36%) nor desirable (14%). Although waiving consent was agreed as desirable for recruitment (77%), the medicine optimization leads were uncertain about acceptability (62%). Consent at higher organizational levels (clinical commissioning group or Sustainability and Transformation Plan) for data access was less acceptable (68%; 36%) or desirable (77%; 64%).

The majority agreed that primary care providers (100%), primary care pharmacists (93%), and medicine optimization leads (92%) should be involved in designing feedback reports to improve primary care prescribing. The panelists considered that clinical commissioning (46%) and PPIE involvement (36%) were less useful.

Feasibility issues

The panel considered the feedback of specific data, such as subgroups of high-risk patients (100%) or excluding patients not targeted by the feedback (86%), as the most useful for changing clinical practice. Less specific data, such as the total number of prescriptions of a medication, was not considered useful (7%).

For an implementation laboratory, randomization levels at the local level (practice or primary care network) were the most ideal (100%; 91%). Randomization by larger areas such as Sustainability and Transformation Plan level was not ideal (63%).

Online dashboards that connected to the electronic health record system were both acceptable (100%) and desirable (100%); dashboards unconnected to the electronic health record system received less support (64%; 79%). Providing multiple paper copies or emailing PDFs of feedback reports were considered neither desirable (2%; 7%) nor acceptable (21%; 43%).

Seven out of eight suggested methods of presenting feedback were agreed as priorities for the implementation laboratory for further effectiveness evaluations. The statements “whether feedback identifying specific behaviors to be changed is more effective” (93%) and “whether different comparators within the reports are more effective” (79%) received the strongest support. Determining “whether asking practitioners to document the implications of changing practice is more effective” was not recommended for testing (43%).

Discussion

We have identified prescribing priorities and elicited key ethical and feasibility considerations to guide the establishment of a primary care prescribing implementation laboratory. All participants agreed that audit and feedback is both useful and important as a method to improve primary care prescribing. There is a notable support to evaluate a wide range of potential enhancements to improve the effects of feedback on prescribing, which collectively could be addressed by sequential trials within an implementation laboratory.

The panel supported action on a range of problems, prioritizing reductions in prescribing of antibiotics and opioids for chronic, non-cancer pain. Both are recognized internationally as public health threats, with respective legacies of antibiotic resistance and dependence [4, 28]. Other problems deemed as highly important, such as prescribing safety indicators, were rated lower as priorities, given existing quality improvement work.

Detailed prescribing data was preferred, specifically patients at high risks of dose escalation and adverse effects, and excluding those patients not targeted by the feedback [20]. Extracting this data requires access to electronic health record systems and data sharing agreements, thereby increasing complexity and hurdles compared to using current freely available English NHS prescribing data such as OpenPrescribing.com [29] and ePACT2 [30]. These initiatives and others providing practice-aggregated prescriptions by medicine category were not helpful for feedback. Medicines with more than one indication may be less suitable for feeding back the total number of prescriptions as exceptions are easier to justify [31]. Highlighting smaller high-priority subgroups of patients may avoid overwhelming practices, causing avoidance and inaction, than all patients prescribed antibiotics or opioids.

Consenting practices through an opt-out approach was agreed to be the most acceptable and desirable. Clinical commissioning group consent or waiving consent completely was desirable, but less acceptable. Waiving consent to reduce participation burden and undermining of results is complicated by the current data protection regulation but may increase participation of practices not routinely involved in primary care research and most likely to benefit. This method was recently used in a UK-wide audit and feedback trial to reduce antibiotic prescribing [32].

The panel recommended randomization in implementation laboratory trials should occur at the practice or primary care network level. Primary care network level may reduce trial arm contamination as practices increasingly work closer together, share staff, and develop local quality improvement initiatives [33]. Practice-level randomization may increase inherent risks of organizational instability for an implementation laboratory where sequential trials are likely to occur over a period of years.

The panel recommended feedback using online dashboards that connect to the electronic health record system, although not currently used by medicine optimization leads. Evidence suggest dashboards providing easily accessible and immediate access to information (e.g., as a screen saver) may improve adherence to recommendations and improve patient outcomes [34]. Multiple dashboards targeting different priorities may cause user fatigue, increase barriers to data access, and without regular prompts may reduce impact [34].

Nearly all suggestions for feedback modifications presented were recommended for further evaluation of effectiveness. This is not unexpected given the lack of progress in the understanding of audit and feedback mechanisms of actions or the key “active ingredients” to produce change [13].

There were three main study limitations. Firstly, Delphi consensus processes can produce but not guarantee the validity of an agreed set of endorsements. For our purposes, given uncertainties around the optimal design of an audit and feedback implementation laboratory, we leveraged a reasonably diverse range of stakeholder perspectives and attained consensus for most statements within two rounds, which lends legitimacy to our future plans.

Secondly, despite our best efforts, we did not achieve a larger sample size. Less than a fifth of people invited participated in the consensus process, raising the possibility that our respondents had unusually high interests in audit and feedback or research and those not participating felt unable to contribute expertise in this area. The lowest response was from the audit and feedback “Metalab” mailing list of researchers (10% of those invited). We surmise two possible reasons why this may be: Firstly, the mailing list includes audit and feedback researchers; however, it is also open to those interested in audit and feedback more generally (e.g., quality improvement staff) who may not have considered themselves to be experts and therefore did not feel the study was appropriate for them. Secondly, the researchers and experts on the mailing list have been collaborators in our previous and planned work and considered themselves insufficiently distanced. The response rates for medicine optimization leads and patients and public group (33% and 40%, respectively) are more representative of survey responses generally. We were unable to collect information on non-responders to determine non-responder bias. We at least avoided any potential attrition bias by ensuring complete participation in both rounds. Moreover, we still achieved a reasonable overall balance of participants.

Finally, we did not seek the views of primary care providers in this study, although such views were represented in earlier work and within the audit and feedback researchers, medicine optimization leads, and study team. We did gain international perspectives from audit and feedback researchers with emerging experience of developing implementation laboratories in other healthcare contexts. Our patient and public participants also had an experience of oversight for previous trials of audit and feedback in primary care [19] and helped ensure that plans for an implementation laboratory took account of public priorities. Future work should aim to involve end-users, including physicians, in the intervention and implementation laboratory design.

The key endorsements for developing a primary care prescribing implementation laboratory concern the need for specific prescribing data, offering feedback at the lowest aggregate level, using opt-out consent for practices (although preferably waived), and developing online dashboards connected to electronic health record systems. These endorsements fit with those suggested by Brehaut et al. to optimize the effectiveness of audit and feedback [35], although some received stronger support than others. Most methods of presenting feedback were recommended for effectiveness evaluation, reflecting continuing uncertainty on how best to optimize audit and feedback in routine service development. Different variants of audit and feedback, such as using an average- or high-performing comparator, have little or no cost implications. This suggests that marginal gains in feedback effects, such as an additional 1% in effectiveness (from 4 to 5%) in problematic prescribing, are likely to be worthwhile at a population level and would be relatively straightforward to test within an implementation laboratory [21].

Conclusions

This small study has produced expert consensus-based endorsements for the optimal design of an audit and feedback implementation laboratory addressing high-priority prescribing issues in UK primary care. Any successful collaboration will depend on identifying shared priorities and addressing practical and ethical considerations. An implementation laboratory at scale may lead to the development of the first national clinical audit to implement radical incrementalism to improve effectiveness.