A Retrospective, Longitudinal External Study of the robustness and reproducibility of National Antibacterial Prescribing Survey Data

Background Point prevalence surveys are used internationally to audit antibacterial use as well as the impact of interventions on improving prescribing and resistance rates. The annual National Antibacterial Prescribing Survey provides data on the appropriateness of antibacterial agent prescribing in Australian hospitals. Assessing the survey’s robustness and result reproducibility is essential to its role in improving antibacterial prescribing practice. Aim To evaluate the reproducibility of internal assessments of antibacterial agent prescribing of both guideline compliance and appropriateness from a Western Australian hospital. Method Census data of 1051 prescriptions from 2013 to 2017 surveys were independently assessed for compliance based on Australian Therapeutic Guidelines - Antibiotics, and appropriateness, based on agent selection, therapy duration and microbiological test results. Concordance of these findings with internal hospital assessments was analysed. Results This external study did not reproduce internal hospital audit results for compliance with guideline parameters. Non-compliant prescribing rate was significantly (p < 0.001) higher externally at 50.7% (533/1051) than internal assessment at 34.9% (367/1051). External analysis also found a significantly smaller proportion of prescriptions to be appropriate (551/1051, 52.4%) compared to internal analysis (745/1051, 70.9%) p < 0.001. Cohen’s Kappa analysis found a moderate agreement for compliance (0.49) and appropriateness (0.50) between the external and internal evaluations. Conclusion The lack of adequate reproducibility of compliance and appropriateness assessments may limit the generalisability of the audit’s results. Validating point prevalence surveys that assess antibacterial agent prescribing can increase confidence and improve reproducibility of their findings; as they provide important data for antimicrobial stewardship programs. Supplementary information The online version contains supplementary material available at 10.1007/s11096-022-01411-w.


Introduction
Increased global prevalence of antibacterial resistance (AR) due to inappropriate antibacterial agent prescribing and

Impact statements
• To reduce the risk of antimicrobial resistance, hospital antibacterial prescribing should be optimal and based on evidence-based guidelines. • Audit tools, such as the National Antimicrobial Prescribing Survey (NAPS) in Australia, are useful to monitor antimicrobial stewardship.

Aim
This study aimed to externally validate the level of compliance with guidelines and additionally appropriateness of antibacterial agent prescribing at a large teaching hospital in Western Australia (WA). Concordance between the internal and external evaluations and data reproducibility were also evaluated.

Ethics approval
The WA Health Governance, Evidence, Knowledge and Outcome system (GEKO) reviewed and approved this study as a Quality Improvement Activity (study number 18,599

Study design
A retrospective, repeated cross-sectional, observational study analysed hospital NAPS; pre-collected in November of each year from 2013 to 2016, and in October of 2017.
The data sets included all antibacterial agents prescribed at 8:00am on the day of the survey, as well as any surgical prophylaxis or single doses administered in the previous 24 h. Data were collected by hospital pharmacists alongside infectious diseases clinicians at a 450 bed adult teaching hospital in WA.

Data collection
Raw NAPS data, collected on NAPS data collection forms (supplementary material) [15] were obtained as Excel sheets and combined into a single dataset before applying data exclusions; only agents administered via the parenteral, oral and intraperitoneal routes were included; topical, inhaled and vaginal antibacterials, as well as antifungals and antivirals were excluded. Duplicate entries were removed.

Data analysis
Compliance was assessed using the Australian Therapeutic Guidelines -Antibiotic (Version 14 (2010) [21] for 2013 and 2014 data; Version 15 (2015) [16] for 2015-2017), whilst prescribing appropriateness was assessed based on NAPS guidelines (supplementary material). Compliance was based on the recorded indication for prescribing, agent selection, dosage, frequency, route of administration and duration of usage has significantly increased rates and average length of hospitalisation, patient morbidity, mortality and associated societal and healthcare costs [1][2][3][4][5]. Antimicrobial Stewardship (AMS) aims to improve antimicrobial agent use to optimise patient outcomes and reduce adverse effects and AR.
As part of AMS, point prevalence surveys (PPS) may be used for assessing the quality of antimicrobial use in the hospital setting, and to inform interventions for its improvement. Repetition of validated PPS offers a reliable overview of trends and the impact of applied interventions on antimicrobial use [6][7][8][9]. Different countries utilise PPS to evaluate and report on the appropriateness of their antimicrobial consumption [10,11]. A PPS of Australian adult hospital patients reported an inappropriate antibacterial prescribing rate of 47% [12]. Similarly, a Chinese study found 51.4% of prescriptions inappropriate [13] and in American hospitals, up to 50% of antibiotic prescribing was incorrect due to therapy duration, indication or agent choice [14]. In Australia, the National Antimicrobial Prescribing Survey (NAPS), commenced in 2011, and its AMS audit tool has been increasingly used. In 2017, NAPS was undertaken by 314 Australian public and private hospitals, and comprised an analysis of 26,277 antimicrobial prescriptions [15]. Educational tools, online modules and case examples are provided to assist auditors in the completion of the PPS. However, NAPS acknowledges the limitation that these assessments are subjective and dependent upon interpretation by hospital personnel [15]. Assessment of the compliance parameter for the NAPS annual audit is based on the Australian Therapeutic Guidelines -Antibiotics [16], except where local and hospital based guidelines apply. Alternatively, therapy may be directed by infectious diseases clinicians or microbiological test results. Where therapy is based on a diagnosed indication, agent selection, dose, route of administration as well as duration of therapy are assessed for compliance with guidelines for that indication. A separate assessment for appropriateness also considers parameters such as a patient's medical history, local resistance patterns and medication supply issues; which may introduce subjectivity and raise the ambiguity of inter-and intra-rater reliability and result reproducibility.
Concerns regarding scientific data integrity and reproducibility have received increased international interest [17,18]. Reportedly up to 94% of medical study results are not entirely reproducible, with 65% reporting inconsistent results [18,19]. To improve reproducibility, a robust study protocol is required that can standardise the collection and analysis of data variables across replicate studies [19,20]. The NAPS has thus far not been subjected to an external audit process. three pharmacist members of the academic research team produced identical results.

Results
Of 1518 antimicrobial prescriptions, 350 records were excluded (107 topical, inhaled and vaginal; 111 antivirals, 128 antifungals, and four duplicate entries), leaving 1168 records for analysis from which the 15 most frequently prescribed agents were analysed. This data set contained 1051 (90.0%) prescriptions (

Prevalence of antibacterial use
Prior to 2017, for most individual years, piperacillin-tazobactam was the most frequently prescribed antimicrobial agent (mean 25.2%; range 21.0-28.1%), falling to 6.9% in 2017 due to global stock shortages that year [15,25]. Overall, cefazolin was the second most frequently prescribed agent except in 2017 when it was number one (18.8%) ( Table 1).
therapy. Each record was then given an overall classification of "Compliant", "Not compliant", "Directed therapy", "No guidelines available" or "Not assessable". Prescribing appropriateness was categorised as "Appropriate", "Inappropriate", and "Not assessable", based on appropriate agent selection for the identified or likely causative organism, therapy duration, agent spectrum of action, and where available based on microbiological tests.
The Western Australian Therapeutic Advisory Group (WA-TAG) Surgical Antibiotic Prophylaxis Guideline: Adults [22], Australian Medicines Handbook [23] and hospital specific treatment guidelines were consulted where relevant. The external data analysis was performed initially by one clinical pharmacist. Where any doubt existed, a multidisciplinary discussion included three academic pharmacists all with clinical and research experience in infectious diseases and antimicrobial stewardship.
The data set was imported into SPSS Statistics 25 software for statistical analysis (IBM Corp. Armonk NY). The level of agreement between internal and external assessments with respect to compliance and appropriateness parameters were assessed using Cohen's kappa statistic. The one-sample Chi-square statistic was used to assess whether the numbers of prescriptions observed within the 5 categories of compliance for the external assessment were similar to the distribution across these categories observed in the internal assessment. A similar analysis was performed for appropriateness. Intra-rater reliability (for the external assessment) was optimised by a single researcher utilising the same tools and methods throughout assessment; and finally an independent random cross-check of 30 assessments of guideline compliance and appropriateness by the

External versus internal findings
Overall, considerable non-concordance occurred between the external and internal assessments for compliance with guidelines ( Table 2) and appropriateness (Table 3) of prescribing for all indications. Antibacterials were most frequently non-compliant for surgical prophylaxis according to this external study, with 116/128 (90.6%) compared with the internal hospital analysis 94/128 (73.4%). Similarly, As shown in Table 2 common indications included surgical prophylaxis (12.2%), community acquired pneumonia (CAP) (7.8%), skin and soft tissue infections (SSTI) (5.0%) and hospital acquired pneumonia (HAP) (3.8%).

External Guideline Compliance Assessment
This external assessment found 50.7% (n = 533) prescriptions were non-compliant with any guidelines. Of 226 (21.6%) compliant prescriptions; 216 (20.6%) were compliant with Therapeutic Guidelines -Antibiotics and 10 prescriptions (1.0%) were compliant with hospital specific, WA-TAG or Australian Medicines Handbook guidelines ( Table 2). One hundred and ninety one (18.2%) prescriptions were for therapy directed by an infectious diseases clinician or based on microbiological tests.
The highest rates of non-compliance were observed with cefalexin (  between external versus internal analysis were also respectively identified. Cohen's Kappa cross tabulations of the appropriateness parameter (Table 5) revealed an overall agreement between internal and external auditors in 779/1051 (74.1%) of assessments. The largest discrepancy in agreement was in external auditors finding 174/745 (23.4%) of internally "appropriate" prescriptions to be "inappropriate".

Discussion
To the best of our knowledge, this is the first study undertaken in Australia to evaluate NAPS reliability by assessing concordance between hospital internal assessments and an external audit of antimicrobial prescribing.
Globally and in Australian hospitals, the main indications for antimicrobial prescriptions were pneumonia, SSTI and surgical prophylaxis [10,29]. In this study, surgical prophylaxis accounted for 12.2% of antibacterial usage, which was higher than that reported in Canadian hospitals (8.9%) [11] and in Switzerland (9.1%) [27], but more closely aligned with the average five year NAPS rates (13.4%) in Australia [26,28,[30][31][32]. This hospital reported a lower rate of antibacterial prescribing for CAP (7.8%) than national NAPS (11.4%) and Canadian data (14.4%) [11]. Similarly, antibacterial use for the treatment of HAP in this hospital (4.0%) was lower than 9.1% reported in Canada [11]. Pneumonia treatment was the most common indication (19.2%) in a global PPS of 53 countries [10].
Surgical prophylaxis retains one of the highest global rates of non-compliant prescribing with cefazolin being frequently used. It accounted for 64.1% of surgical prophylaxis prescriptions in this hospital, which was similar to rates in Canadian hospitals (69.7%) [11,29]. Overall rates of noncompliant and inappropriate prescribing for this indication were 90.6% and 86.7% respectively, for this external evaluation, which differed significantly (p < 0.001) to internal assessments (73.4% and 62.5% respectively). International and Australian hospitals have reported inappropriate use of external analysis could not replicate rates of non-compliant prescribing for other common indications including CAP, SSTI and HAP ( Table 2). Analysis of the difference between external and internal audits revealed the largest discrepancy in compliance with guidelines to be in the 2015 audit (Table 2).
External assessment found 111/128 (86.7%) of surgical prophylaxis prescriptions were inappropriate, compared to 80/128 (62.5%) reported by internal analysis. Rates of inappropriate prescribing also differed between external and internal analyses for CAP, SSTI, and HAP (Table 3).
In the assessment of the overall five year data, Cohen's Kappa revealed a moderate agreement between internal and external prescription analyses for compliance with guidelines (0.49) and for appropriateness (0.50). Cross tabulations for Cohen's Kappa analysis revealed that agreement between internal and external assessors varied for the different categories of compliance (Table 4) and appropriateness ( Table 5). The largest discrepancy in agreement was in the "Compliant with guidelines" variable; as external assessors agreed with only 182/369 (49.3%) of internal assessments. There was higher agreement in the "Non-compliant with guidelines" variable with 323/367 (88.0%) of internal assignment being reproduced by external assessors. Overall, agreement between internal and external analyses was reached in 665/1051 (63.3%) of assessments for compliance with guidelines (Table 4). Non-compliant prescriptions due to an incorrect dose or frequency of administration were reported at rates of 208 (19.8%) prescriptions by external and 111 (10.6%) by internal assessors. Differences in the assessment of allergy 18 Table 3 Comparison of external audit with internal assessment of hospital National Antimicrobial Prescribing Survey (NAPS) prescribing data for appropriateness of individual agents, indications and audit years guidelines. Documentation of surgical incision and antibacterial administration timing, as well as post-surgical dose requirements may improve reproducibility [29].
The average five year rates of non-compliance with guidelines was higher in this hospital whether by external (50.7%) or internal (34.9%) assessments, compared to the national data (24.8%). The largest discrepancy in non-compliance assessment between external (60.0%) and internal (40.0%) audits was reported in 2015. This was largely due to the update (in August 2015) of the Therapeutic Guidelines -Antibiotics, which changed commonly prescribed antibacterial agents dosage and frequency of administration regimens, but these were not reflected in the internal assessments carried out in November. Inappropriate prescribing was reported similarly as lower rates by internal (27.6%) than external (39.7%) assessments. These were both higher than national rates (22.5%), while international rates ranged between 14.1% and 78.9% [34].
antibacterial agents for surgical prophylaxis due to treatment beyond one day; with rates ranging from 28.0 to 54.2% [11,25,27,29]. In this study, only 18.0% of prescriptions for this indication were written for a single dose of prophylaxis, below 39.7% reported in other studies [11], but higher than 12.7% reported in Switzerland [27].
The guideline of a one day window of appropriate surgical prophylaxis prescribing is not unique to the Australian PPS interpretation [11,33]. However, NAPS indicates this 24 h period is valid where there are no guidelines that direct therapy otherwise. Rather than a single pre-surgical, prophylactic dose, as recommended by the Therapeutic Guidelines -Antibiotics [16], most prescriptions were written for ongoing dosing and thus considered non-compliant by this external assessment. This accounted for most of the discrepancy between the external and internal evaluations for this indication. Internal auditors deemed prescriptions within the 24 h window to be appropriate, regardless of the   other international PPS should provide sufficient robustness to enable acceptably reproducible data to be obtained to reflect hospital antimicrobial prescribing practice; especially since it is an AMS assessment tool for many hospitals. Further research into the reproducibility of NAPS results and exploration of the completeness of documentation of data used in evaluating appropriateness is desirable.

Acknowledgements
The authors would like to acknowledge the assistance of the Hospital Pharmacy's staff during the data collection process. For his assistance in data analysis and dissemination, the authors would also acknowledge the contribution of Dr Richard Parsons, from Curtin University.

Funding
The authors did not receive support from any organisation for the submitted work.

Open Access funding enabled and organized by CAUL and its Member Institutions
Conflicts of interest The authors have no financial or non-financial interests to disclose.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons. org/licenses/by/4.0/.
It can be expected that individual hospital NAPS assessments will vary from the mean pooled national data to some degree, due to geographical variations affecting differences in antibacterial stock availability. Other factors include local hospital guidelines, resistance patterns, private and public sector policy differences and remoteness level of the institution from major cities [10,34]. Similar concerns are expressed in a comparison of international PPS data and their reflection of global antibacterial usage [34]. The hospital in this study was a public hospital in a major city; while NAPS data pools national public and private sector, city, remote and regional hospitals data. Chi square tests indicated that the profile of both compliance and appropriateness differed significantly between internal and external assessments (p < 0.001). Exploring potential reasons for this difference may improve assessment reproducibility. While acknowledging that this audit of a single hospital limits generalisability, a much larger number of hospitals would require investigation to establish findings that would be generalisable. This study is not representative of national NAPS, it does however warrant further, large scale investigations into the need to validate this PPS as an AMS audit tool. Several limitations, including access to only the prescribing data pre-collected by hospital staff, have impacted the scope of this research.
The retrospective design further limited data analysis of 2016 results since the date of prescribing was not recorded, thus appropriateness of duration of therapy was not assessable. Insufficient reporting and missing information on indications and microbiological test results could influence the external audit increasing the level of non-assessable reports. Development of standardised data collection which fills the gaps and justifies decision making is a necessary inclusion to improve NAPS data reproducibility.
It is unclear from the available data why 'no guidelines available' was reported at a higher rate by internal assessors compared to external assessors (11.9% versus 3.0%) which again reinforces the need for additional comments boxes to explain decisions. To improve reproducibility of PPS results, we recommend inter-and intra-rater reliability checks be carried out every year ahead of NAPS. These reliability checks are among the most common initiatives to enhance reproducibility of antimicrobial use data [33,35].

Conclusion
High rates of non-compliant and inappropriate prescribing were identified in this hospital, consistent with national and international concerns regarding antibacterial agent prescribing. This external audit was unable to replicate internal hospital assessments for the studied parameters. NAPS and