Impact statements

  • To reduce the risk of antimicrobial resistance, hospital antibacterial prescribing should be optimal and based on evidence-based guidelines.

  • Audit tools, such as the National Antimicrobial Prescribing Survey (NAPS) in Australia, are useful to monitor antimicrobial stewardship.

  • To minimise ambiguity of the rationale for prescribed antibiotics and promote reproducibility, audit tools should align with national antibiotic prescribing guidelines.

  • Implementation of a commentary section in the NAPS data collection form, to outline reasons for assessment decisions not aligned with guidelines, could improve their reproducibility and impact on improving antimicrobial stewardship.

Introduction

Increased global prevalence of antibacterial resistance (AR) due to inappropriate antibacterial agent prescribing and usage has significantly increased rates and average length of hospitalisation, patient morbidity, mortality and associated societal and healthcare costs [1,2,3,4,5]. Antimicrobial Stewardship (AMS) aims to improve antimicrobial agent use to optimise patient outcomes and reduce adverse effects and AR. As part of AMS, point prevalence surveys (PPS) may be used for assessing the quality of antimicrobial use in the hospital setting, and to inform interventions for its improvement. Repetition of validated PPS offers a reliable overview of trends and the impact of applied interventions on antimicrobial use [6,7,8,9].

Different countries utilise PPS to evaluate and report on the appropriateness of their antimicrobial consumption [10, 11]. A PPS of Australian adult hospital patients reported an inappropriate antibacterial prescribing rate of 47% [12]. Similarly, a Chinese study found 51.4% of prescriptions inappropriate [13] and in American hospitals, up to 50% of antibiotic prescribing was incorrect due to therapy duration, indication or agent choice [14]. In Australia, the National Antimicrobial Prescribing Survey (NAPS), commenced in 2011, and its AMS audit tool has been increasingly used. In 2017, NAPS was undertaken by 314 Australian public and private hospitals, and comprised an analysis of 26,277 antimicrobial prescriptions [15]. Educational tools, online modules and case examples are provided to assist auditors in the completion of the PPS. However, NAPS acknowledges the limitation that these assessments are subjective and dependent upon interpretation by hospital personnel [15]. Assessment of the compliance parameter for the NAPS annual audit is based on the Australian Therapeutic Guidelines - Antibiotics [16], except where local and hospital based guidelines apply. Alternatively, therapy may be directed by infectious diseases clinicians or microbiological test results. Where therapy is based on a diagnosed indication, agent selection, dose, route of administration as well as duration of therapy are assessed for compliance with guidelines for that indication. A separate assessment for appropriateness also considers parameters such as a patient’s medical history, local resistance patterns and medication supply issues; which may introduce subjectivity and raise the ambiguity of inter- and intra-rater reliability and result reproducibility.

Concerns regarding scientific data integrity and reproducibility have received increased international interest [17, 18]. Reportedly up to 94% of medical study results are not entirely reproducible, with 65% reporting inconsistent results [18, 19]. To improve reproducibility, a robust study protocol is required that can standardise the collection and analysis of data variables across replicate studies [19, 20]. The NAPS has thus far not been subjected to an external audit process.

Aim

This study aimed to externally validate the level of compliance with guidelines and additionally appropriateness of antibacterial agent prescribing at a large teaching hospital in Western Australia (WA). Concordance between the internal and external evaluations and data reproducibility were also evaluated.

Ethics approval

The WA Health Governance, Evidence, Knowledge and Outcome system (GEKO) reviewed and approved this study as a Quality Improvement Activity (study number 18,599). Reciprocal ethics approval was provided by Curtin University Human Research Ethics Committee (HRE2017-0843; date approved: 05/12/2017).

Method

Study design

A retrospective, repeated cross-sectional, observational study analysed hospital NAPS; pre-collected in November of each year from 2013 to 2016, and in October of 2017. The data sets included all antibacterial agents prescribed at 8:00am on the day of the survey, as well as any surgical prophylaxis or single doses administered in the previous 24 h. Data were collected by hospital pharmacists alongside infectious diseases clinicians at a 450 bed adult teaching hospital in WA.

Data collection

Raw NAPS data, collected on NAPS data collection forms (supplementary material) [15] were obtained as Excel sheets and combined into a single dataset before applying data exclusions; only agents administered via the parenteral, oral and intraperitoneal routes were included; topical, inhaled and vaginal antibacterials, as well as antifungals and antivirals were excluded. Duplicate entries were removed.

Data analysis

Compliance was assessed using the Australian Therapeutic Guidelines - Antibiotic (Version 14 (2010) [21] for 2013 and 2014 data; Version 15 (2015) [16] for 2015–2017), whilst prescribing appropriateness was assessed based on NAPS guidelines (supplementary material). Compliance was based on the recorded indication for prescribing, agent selection, dosage, frequency, route of administration and duration of therapy. Each record was then given an overall classification of “Compliant”, “Not compliant”, “Directed therapy”, “No guidelines available” or “Not assessable”. Prescribing appropriateness was categorised as “Appropriate”, “Inappropriate”, and “Not assessable”, based on appropriate agent selection for the identified or likely causative organism, therapy duration, agent spectrum of action, and where available based on microbiological tests.

The Western Australian Therapeutic Advisory Group (WA-TAG) Surgical Antibiotic Prophylaxis Guideline: Adults [22], Australian Medicines Handbook [23] and hospital specific treatment guidelines were consulted where relevant. The external data analysis was performed initially by one clinical pharmacist. Where any doubt existed, a multidisciplinary discussion included three academic pharmacists all with clinical and research experience in infectious diseases and antimicrobial stewardship.

The data set was imported into SPSS Statistics 25 software for statistical analysis (IBM Corp. Armonk NY). The level of agreement between internal and external assessments with respect to compliance and appropriateness parameters were assessed using Cohen’s kappa statistic. The one-sample Chi-square statistic was used to assess whether the numbers of prescriptions observed within the 5 categories of compliance for the external assessment were similar to the distribution across these categories observed in the internal assessment. A similar analysis was performed for appropriateness. Intra-rater reliability (for the external assessment) was optimised by a single researcher utilising the same tools and methods throughout assessment; and finally an independent random cross-check of 30 assessments of guideline compliance and appropriateness by the three pharmacist members of the academic research team produced identical results.

A p-value < 0.05 indicated a statistically significant association for the Chi-square tests. The interpretation of Cohen’s kappa statistic was as follows: <0.40 indicated “poor to fair agreement”; 0.40–0.60 “moderate agreement”; 0.60–0.80 “substantial agreement” and > 0.80 “excellent agreement” [24].

Results

Of 1518 antimicrobial prescriptions, 350 records were excluded (107 topical, inhaled and vaginal; 111 antivirals, 128 antifungals, and four duplicate entries), leaving 1168 records for analysis from which the 15 most frequently prescribed agents were analysed. This data set contained 1051 (90.0%) prescriptions (Table 1), prescribed for 645 (61.4%) males, 405 (38.5%) females and one (0.1%) other. The excluded 10.0% (n = 117) of prescriptions comprised 29 different antibacterial agents; each of which was prescribed ≤ 14 times (median 3). The top 15 agents were each prescribed, on average > 70 times (median 55).

Table 1 Frequency of prescribing of the top 15 antibacterial agent prescriptions on the annual day of the National Antimicrobial Prescribing Survey (NAPS) for the period 2013 to 2017
Full size table

Prevalence of antibacterial use

Prior to 2017, for most individual years, piperacillin-tazobactam was the most frequently prescribed antimicrobial agent (mean 25.2%; range 21.0–28.1%), falling to 6.9% in 2017 due to global stock shortages that year [15, 25]. Overall, cefazolin was the second most frequently prescribed agent except in 2017 when it was number one (18.8%) (Table 1).

As shown in Table 2 common indications included surgical prophylaxis (12.2%), community acquired pneumonia (CAP) (7.8%), skin and soft tissue infections (SSTI) (5.0%) and hospital acquired pneumonia (HAP) (3.8%).

Table 2 Comparison of external audit with internal assessment of hospital National Antimicrobial Prescribing Survey (NAPS) prescribing data for compliance with guidelines of individual agents, indications and audit years
Full size table

External Guideline Compliance Assessment

This external assessment found 50.7% (n = 533) prescriptions were non-compliant with any guidelines. Of 226 (21.6%) compliant prescriptions; 216 (20.6%) were compliant with Therapeutic Guidelines - Antibiotics and 10 prescriptions (1.0%) were compliant with hospital specific, WA-TAG or Australian Medicines Handbook guidelines (Table 2). One hundred and ninety one (18.2%) prescriptions were for therapy directed by an infectious diseases clinician or based on microbiological tests.

The highest rates of non-compliance were observed with cefalexin (28/35; 80.0%), cefazolin (128/165; 77.6%), piperacillin-tazobactam (116/222; 52.3%) and metronidazole (42/78; 53.8%). The highest rates of compliance with guidelines were seen for ceftriaxone (39/80; 48.8%) and flucloxacillin (31/72; 43.1%) (Table 2).

External Appropriateness Assessment

Although more than half (551/1051; 52.4%) of antibacterial prescriptions were deemed appropriate (Table 3), overall, 39.7% (417) of prescriptions were inappropriate; including 44 (4.2%) prescriptions which were not required according to their recorded indication.

Table 3 Comparison of external audit with internal assessment of hospital National Antimicrobial Prescribing Survey (NAPS) prescribing data for appropriateness of individual agents, indications and audit years
Full size table

The most commonly prescribed antibacterial agent, piperacillin-tazobactam was inappropriately prescribed in 84/222 (37.8%) cases. Cefazolin was the most frequently deemed inappropriately prescribed antibacterial (107/165; 64.8%) (Table 3). Agents with high rates of appropriate prescribing included flucloxacillin (54/72; 75.0%), meropenem (40/55; 72.7%) and vancomycin (41/57; 71.7%); the latter two agents were also frequently prescribed as directed therapy at rates of 58.2% (32/55) and 57.9% (33/57), respectively.

External versus internal findings

Overall, considerable non-concordance occurred between the external and internal assessments for compliance with guidelines (Table 2) and appropriateness (Table 3) of prescribing for all indications. Antibacterials were most frequently non-compliant for surgical prophylaxis according to this external study, with 116/128 (90.6%) compared with the internal hospital analysis 94/128 (73.4%). Similarly, external analysis could not replicate rates of non-compliant prescribing for other common indications including CAP, SSTI and HAP (Table 2). Analysis of the difference between external and internal audits revealed the largest discrepancy in compliance with guidelines to be in the 2015 audit (Table 2).

External assessment found 111/128 (86.7%) of surgical prophylaxis prescriptions were inappropriate, compared to 80/128 (62.5%) reported by internal analysis. Rates of inappropriate prescribing also differed between external and internal analyses for CAP, SSTI, and HAP (Table 3).

External findings could not replicate internal compliance and appropriateness assessments for the different antibacterial agents prescribed. External assessors found 116/222 (52.3%) of piperacillin-tazobactam and 128/165 (77.6%) of cefazolin prescriptions were non-compliant with guidelines; compared to 91/222 (41.0%) and 95/165 (57.6%) found by internal assessments respectively (Table 2). Inappropriate prescribing was reported by external assessment as 83/222 (37.4%) for piperacillin-tazobactam and 106/165 (64.2%) of cefazolin prescriptions; while internal assessment found lower rates of 57/222 (25.7%) and 64/165 (38.8%) respectively (Table 3).

More prescriptions were deemed ‘Not Assessable’ by external auditors for compliance (69/1051; 6.6%) (Table 2) and appropriateness (83/1051; 7.9%) (Table 3) parameters compared to internal assessors (30/1051; 2.9%) and (16/1051; 1.6%) respectively. This external audit reported fewer ‘No Guidelines Available’ assessments (32/1051; 3.0%) in the compliance parameter than internal assessors (125/1051; 11.9%) (Table 2).

In the assessment of the overall five year data, Cohen’s Kappa revealed a moderate agreement between internal and external prescription analyses for compliance with guidelines (0.49) and for appropriateness (0.50). Cross tabulations for Cohen’s Kappa analysis revealed that agreement between internal and external assessors varied for the different categories of compliance (Table 4) and appropriateness (Table 5). The largest discrepancy in agreement was in the “Compliant with guidelines” variable; as external assessors agreed with only 182/369 (49.3%) of internal assessments. There was higher agreement in the “Non-compliant with guidelines” variable with 323/367 (88.0%) of internal assignment being reproduced by external assessors. Overall, agreement between internal and external analyses was reached in 665/1051 (63.3%) of assessments for compliance with guidelines (Table 4). Non-compliant prescriptions due to an incorrect dose or frequency of administration were reported at rates of 208 (19.8%) prescriptions by external and 111 (10.6%) by internal assessors. Differences in the assessment of allergy 18 (1.7%) versus 11 (1.0%) and microbiology 36 (3.4%) versus 23 (2.2%) prescribing mismatch between external versus internal analysis were also respectively identified.

Table 4 Cohen’s Kappa cross tabulation for agreement of compliance with guidelines variable between internal and external analyses
Full size table
Table 5 Cohen’s Kappa cross tabulation for agreement of appropriateness variable between internal and external analyses
Full size table

Cohen’s Kappa cross tabulations of the appropriateness parameter (Table 5) revealed an overall agreement between internal and external auditors in 779/1051 (74.1%) of assessments. The largest discrepancy in agreement was in external auditors finding 174/745 (23.4%) of internally “appropriate” prescriptions to be “inappropriate”.

Discussion

To the best of our knowledge, this is the first study undertaken in Australia to evaluate NAPS reliability by assessing concordance between hospital internal assessments and an external audit of antimicrobial prescribing.

Antibacterial agents prescribed in this hospital showed similar trends as national [26] and international [10, 11, 27] PPS. The most frequently prescribed antibacterial agents were penicillin-beta lactamase inhibitor combinations and first and third generation cephalosporins. Piperacillin-tazobactam prescribing (28.1%) was higher than national NAPS (6.3%)[28] and international PPS (7.7%)[10] data in 2015 and most other years. Higher overall rates of non-compliance (50.7%) than national NAPS (24.8%)[28] and international data [10] were identified. A global PPS of 53 countries reported rates of compliant prescribing of 77.4% [10].

Globally and in Australian hospitals, the main indications for antimicrobial prescriptions were pneumonia, SSTI and surgical prophylaxis [10, 29]. In this study, surgical prophylaxis accounted for 12.2% of antibacterial usage, which was higher than that reported in Canadian hospitals (8.9%)[11] and in Switzerland (9.1%)[27], but more closely aligned with the average five year NAPS rates (13.4%) in Australia [26, 28, 30,31,32]. This hospital reported a lower rate of antibacterial prescribing for CAP (7.8%) than national NAPS (11.4%) and Canadian data (14.4%)[11]. Similarly, antibacterial use for the treatment of HAP in this hospital (4.0%) was lower than 9.1% reported in Canada [11]. Pneumonia treatment was the most common indication (19.2%) in a global PPS of 53 countries [10].

Surgical prophylaxis retains one of the highest global rates of non-compliant prescribing with cefazolin being frequently used. It accounted for 64.1% of surgical prophylaxis prescriptions in this hospital, which was similar to rates in Canadian hospitals (69.7%) [11, 29]. Overall rates of non-compliant and inappropriate prescribing for this indication were 90.6% and 86.7% respectively, for this external evaluation, which differed significantly (p < 0.001) to internal assessments (73.4% and 62.5% respectively). International and Australian hospitals have reported inappropriate use of antibacterial agents for surgical prophylaxis due to treatment beyond one day; with rates ranging from 28.0 to 54.2% [11, 25, 27, 29]. In this study, only 18.0% of prescriptions for this indication were written for a single dose of prophylaxis, below 39.7% reported in other studies [11], but higher than 12.7% reported in Switzerland [27].

The guideline of a one day window of appropriate surgical prophylaxis prescribing is not unique to the Australian PPS interpretation [11, 33]. However, NAPS indicates this 24 h period is valid where there are no guidelines that direct therapy otherwise. Rather than a single pre-surgical, prophylactic dose, as recommended by the Therapeutic Guidelines - Antibiotics [16], most prescriptions were written for ongoing dosing and thus considered non-compliant by this external assessment. This accounted for most of the discrepancy between the external and internal evaluations for this indication. Internal auditors deemed prescriptions within the 24 h window to be appropriate, regardless of the guidelines. Documentation of surgical incision and antibacterial administration timing, as well as post-surgical dose requirements may improve reproducibility [29].

The average five year rates of non-compliance with guidelines was higher in this hospital whether by external (50.7%) or internal (34.9%) assessments, compared to the national data (24.8%). The largest discrepancy in non-compliance assessment between external (60.0%) and internal (40.0%) audits was reported in 2015. This was largely due to the update (in August 2015) of the Therapeutic Guidelines - Antibiotics, which changed commonly prescribed antibacterial agents dosage and frequency of administration regimens, but these were not reflected in the internal assessments carried out in November. Inappropriate prescribing was reported similarly as lower rates by internal (27.6%) than external (39.7%) assessments. These were both higher than national rates (22.5%), while international rates ranged between 14.1% and 78.9% [34].

It can be expected that individual hospital NAPS assessments will vary from the mean pooled national data to some degree, due to geographical variations affecting differences in antibacterial stock availability. Other factors include local hospital guidelines, resistance patterns, private and public sector policy differences and remoteness level of the institution from major cities [10, 34]. Similar concerns are expressed in a comparison of international PPS data and their reflection of global antibacterial usage [34]. The hospital in this study was a public hospital in a major city; while NAPS data pools national public and private sector, city, remote and regional hospitals data. Chi square tests indicated that the profile of both compliance and appropriateness differed significantly between internal and external assessments (p < 0.001). Exploring potential reasons for this difference may improve assessment reproducibility. While acknowledging that this audit of a single hospital limits generalisability, a much larger number of hospitals would require investigation to establish findings that would be generalisable. This study is not representative of national NAPS, it does however warrant further, large scale investigations into the need to validate this PPS as an AMS audit tool. Several limitations, including access to only the prescribing data pre-collected by hospital staff, have impacted the scope of this research.

The retrospective design further limited data analysis of 2016 results since the date of prescribing was not recorded, thus appropriateness of duration of therapy was not assessable. Insufficient reporting and missing information on indications and microbiological test results could influence the external audit increasing the level of non-assessable reports. Development of standardised data collection which fills the gaps and justifies decision making is a necessary inclusion to improve NAPS data reproducibility.

It is unclear from the available data why ‘no guidelines available’ was reported at a higher rate by internal assessors compared to external assessors (11.9% versus 3.0%) which again reinforces the need for additional comments boxes to explain decisions. To improve reproducibility of PPS results, we recommend inter- and intra-rater reliability checks be carried out every year ahead of NAPS. These reliability checks are among the most common initiatives to enhance reproducibility of antimicrobial use data [33, 35].

Conclusion

High rates of non-compliant and inappropriate prescribing were identified in this hospital, consistent with national and international concerns regarding antibacterial agent prescribing. This external audit was unable to replicate internal hospital assessments for the studied parameters. NAPS and other international PPS should provide sufficient robustness to enable acceptably reproducible data to be obtained to reflect hospital antimicrobial prescribing practice; especially since it is an AMS assessment tool for many hospitals. Further research into the reproducibility of NAPS results and exploration of the completeness of documentation of data used in evaluating appropriateness is desirable.