Introduction

Adverse drug reactions (ADR) pose a significant burden to healthcare, accounting for 3 to 5% of hospital admissions [1, 2]. It is estimated that 10–17% of hospitalised patients experience an ADR, which may result in up to a two-fold increase in the length of hospital stay [1, 3, 4]. Severe ADRs may have a long-lasting physical and psychological impact on patients [5, 6]. Medication-related hospital admissions cost AU $1.2 billion annually, yet a substantial proportion of ADRs are preventable with appropriate management [7].

An important element of ADR management within a hospital setting is the documentation and reporting of ADR episodes, facilitated by a centralised internal review process, with subsequent reporting to pharmacovigilance authorities. Main aspects of ADR reporting that are often discussed are the under-reporting of ADRs and ways to increase the quantity of reports. However, the quality of ADR reports is equally important as a minimum dataset is required to assess medication reaction causality. This process requires the application of technical skills and knowledge by reporting healthcare professionals, including knowledge of ADR syndromes and pathogenesis, recognition of underlying comorbidities and concurrent conditions mimicking an ADR, familiarity with pharmacologic profiles of medications and drug interactions, and ability to construct a relevant medication timeline [8]. An incomplete or a low-quality report undermines the strength of the association between a medication and a reaction. Comprehensive and accurate risk communication, with recommendations to patients and healthcare professionals is not possible without a high-quality assessment [9].

It is recognised that ADR reporting processes are subject to large practice variations, often influenced by healthcare providers’ attitudes and beliefs, knowledge and skills, clinical expertise and practice settings, as well as familiarity with local protocols and reporting requirements by authorities [10]. Such unintended variations potentially compromise patient safety and diminish the quality of care. Appropriate clinical indicators to measure and monitor the quality in ADR management processes within hospital settings remain ill-defined. Moreover, currently available tools that assess the quality of ADR reports are almost exclusively designed for use in the research setting [10]. A rapid and reliable assessment tool for pragmatic application in daily clinical settings within hospital environments would facilitate quality monitoring and process improvement.

In this quality improvement initiative, we developed a pragmatic scoring system, Adverse drug reactions reports QUality Algorithm (AQUA-12), that enables regular monitoring of the quality of ADR reporting processes in hospital settings. The primary aim was to evaluate the performance of AQUA-12 during the development and application phases. Secondary aims were to evaluate the quality of ADR reports and to determine potential factors contributing to low-quality reports to target improvements.

Methodology

Setting

AQUA-12 was designed by members of the Adverse Drug Reaction Review Committee (ADRRC) at Alfred Health in Melbourne, Australia. Alfred Health is a quaternary, university-affiliated health institution that provides, among many clinical services, specialist care in transplantation, human immunodeficiency virus (HIV) infection, cystic fibrosis, haemophilia, trauma and burns. The ADRRC is a multidisciplinary group consisting of senior pharmacists in medication safety and medicines information, and specialist physicians in allergy/immunology, dermatology, clinical pharmacology, infectious diseases and internal medicine. Around 200 ADR episodes per year are reported to ADRRC. The committee meets every 2 weeks to discuss ADR reports, assign causality, organise further referrals as required (e.g. allergy services) and provide recommendations regarding future medication management. All healthcare professionals within the institution are encouraged to submit ADR reports; approximately 85% of the reports are submitted by hospital pharmacists [11]. The reporting system is predominantly via an electronic form embedded in the electronic medical record (EMR).

The need to design a scoring system to monitor the quality of ADR reports was first identified in mid-2021 when ADRRC began to develop an ADR education program for hospital pharmacists and junior doctors. This sought to improve the knowledge, technical skills and competency required to conduct a comprehensive assessment of an ADR episode. The tool was intended to assess the completeness of information in submitted ADR reports and would be a surrogate marker to assess practical knowledge and technical attributes. Hence, we intend to measure improvement over time, following the planned educational program.

Development phase

The primary objectives of AQUA-12 were to assess the completeness of data to allow the ADRRC to assess causality and provide effective risk communication to patients. The emphasis was placed on the following data elements for scoring: (i) previous ADR history, (ii) diagnosis or description of actual ADR, (iii) description of key events concerning ADR (i.e. the narrative), (iv) list of suspected medications, (v) consideration of medication timeline relevant to the nature of ADR, (vi) management of ADR episode and (vii) outcome/sequelae. The rationale behind the inclusion of each data element is summarised in Table 1. The data elements also reflect data fields in the ADR reporting form that are required to be completed by healthcare professionals and excluding those automatically populated by the EMR, such as patient details, reporter details. Each data element is assigned a maximum score of 2, except for the relevant medication timeline (temporality) and management, which are assigned a score of 1 each, giving a total score of 12.

Table 1 Included data elements in AQUA-12 quality assessment tool and underlying principles
Full size table

In Round 1 of this cross-sectional study, the first version of AQUA-12 was evaluated using 20 randomly selected ADR reports submitted to ADRRC between 11th January and 4th June 2021. Two ADRRC members (AKA, LG) and one junior doctor, who was not part of the ADRRC, independently assessed the reports retrospectively for completeness after the ADRRC review had been conducted. The scores were then analysed for inter-rater correlation. Any reports displaying discrepancy in total scores between assessors by more than two points were identified and reasons behind differences were discussed. The scoring criteria and wording were refined to make them more concise and easily interpretable.

In Round 2, a revised version of AQUA-12 was retrospectively and independently evaluated using the same 20 ADR reports as above, but by a different set of ADRRC members consisting of one clinical pharmacologist (BS), one dermatologist (MG) and one allergist/immunologist (CZ). External to the ADRRC, a clinical pharmacologist and a junior doctor who were not familiar with the current ADRRC review processes, also independently evaluated the reports. Inter-rater correlation analysis was conducted using the scores of 5 assessors from Round 2. The tool was then further refined to improve functionality.

In both rounds during the development phase, final ADR diagnoses and management recommendations by ADRRC, as well as further clinical information, were made available to the independent assessors scoring the reports.

The final version of AQUA-12 derived from the above process is provided in Table 6.

Evaluation phase

In this phase, AQUA-12 was used to assess 50 consecutive reports submitted to ADRRC between 1st Jan 2022 to 18th April 2022. Reports were scored independently by AKA and LG, and inter-rater correlation analysis was conducted. The first assessor (AKA) prospectively scored the ADR reports in a blinded manner prior to the scheduled ADRRC review fortnightly. The second assessor (LG) independently scored after further information (diagnosis, investigations and recommendations) was made available post-ADRRC review of the reports.

Data variables and outcomes

The following data variables concerning all ADR reports were extracted from electronic medical records: vocation of reporters, treating clinical unit, reaction type, reaction severity and implicated medication classes. Reaction types and implicated medication classes were classified according to the methodology previously described in a publication by ADRRC to maintain consistency [11]. Outcomes of interest were as follows: (i) inter-rater correlation of AQUA-12 scores in both rounds of the development phase and in the prospective evaluation phase, (ii) proportion of high-quality reports using AQUA-12 tool and (iii) factors that may be associated with low-quality reports.

Data analysis

Summary statistics for discrete variables are presented as counts and proportions. Inter-rater correlation analysis results are presented as intraclass correlation coefficient (Cronbach’s alpha) with 95% confidence intervals. Univariable analyses were conducted to identify any factors that may be associated with the poor quality of reports. For differences in proportions between groups, Fisher’s exact or chi-square tests were conducted, and statistically significant results are presented as a two-tailed p value of < 0.05. Data analysis was done using SPSS version 28 (IBM Corporation, Armonk, NY, USA).

Ethics approval

Approval to conduct this study as a low-risk research project was granted by the Alfred Health Human Research and Ethics Committee (project number: 726/21).

Results

A total of 70 ADR reports were included in the final analysis: 20 from the development phase and 50 from the prospective evaluation phase. The characteristics of ADR reports are displayed in Table 2. Most reports were submitted by clinical pharmacists and originated within medical units. Immediate hypersensitivity reactions accounted for one in five reports, and delayed hypersensitivity reactions constituted 60% of all reactions. The majority of reactions reported were moderate to severe in nature, involving predominantly antimicrobials. There were no statistically significant differences in ADR report characteristics between the two phases.

Table 2 ADR report characteristics
Full size table

Table 3 shows the results of interrater correlation analysis from both the development and prospective evaluation phases. AQUA-12 yielded a high degree of correlation among assessors with Cronbach’s alpha value of ≥ 0.90 in all rounds of evaluation. Detailed results on mean scores by individual assessors and inter-item correlation matrices are provided in Tables 7, 8, and 9.

Table 3 Inter-rater correlation analysis for development and prospective evaluation phases of AQUA-12
Full size table

Overall, 52 (74.3%) of reports were of high quality (defined as score of ≥ 10) with 30 (42.9%) reports scoring a possible maximum score of 12 (Table 4). ‘Suspected medication’ (37.1%), ‘Description of key events’ (35.7%) and ‘Actual reaction’ (27.1%) were data elements that were less frequently completed or partially completed.

Table 4 Quality of ADR reports and breakdown of incomplete components
Full size table

Table 5 compares the characteristics between high- and low-quality ADR reports as determined by the AQUA-12 tool. Statistically significant differences in proportions of mild or undocumented severity of reactions were noted between high- and low-quality reports (5.8% high quality vs. 22.2% low quality for mild severity reactions, and 1.9% high quality vs. 22.2% low quality for reactions of undocumented severity). No notable differences exist between reporter and reaction types between the high- and low-quality ADR reports.

Table 5 Differences in characteristics between high quality (AQUA-12 score ≥ 10) and lower quality (score < 10) reports
Full size table
Table 6 Final version of Adverse drug reactions reports QUality Algorithm (AQUA-12) quality assessment tool. ADR Reports Quality Algorithm (AQUA12) version 2, 16.08.21
Full size table
Table 7 Inter-item correlation matrix for development phase, round 1, of evaluation of AQUA-12 tool
Full size table
Table 8 Inter-item correlation matrix for development phase, round 2, of evaluation of AQUA-12 tool
Full size table
Table 9 Inter-item correlation matrix for prospective evaluation phase, round 3, of evaluation of AQUA-12 tool
Full size table

Discussion

In the present study, we developed a tool that can be routinely utilised in hospital settings to monitor the quality and completeness of ADR reports. AQUA-12 was found to be highly reliable among independent assessors. Using AQUA-12, we found that the majority of ADR reports are of high quality. However, important details were missing in a large proportion of reports for certain data elements, such as the ADR narrative, and the description of the actual reaction, which may impact on causality assessment and further management recommendations. AQUA-12 further underscored that the identification of suspected medications is an area that requires further attention for improvement.

A recent review highlighted that large variations exist for ADR management and reporting in hospital settings [10]. It also found that current quality assessment tools for ADR reports are not standardised across the board and are also unnecessarily complex with requirements for a vast array of data. The most widely used criteria, VigiGrade, also consists of a scoring system that is counterintuitive to causality assessment, where points can be assigned to medications started after the onset of the reaction [12]. Furthermore, all available tools have been utilised only in research settings at pharmacovigilance centres. To date, there is no quality assessment tool that is both pragmatic and easily applicable to daily clinical practice in local hospital settings.

Quality improvement methods utilise research tools where process and outcome measures can be monitored in a frequent manner to rapidly detect changes in trends, variations and process limits over time [13]. Adhering to this principle, the ease of application of AQUA-12 provides a distinct advantage, compared to other methods such as questionnaires and surveys, by being able to systematically and reliably monitor ADR reports on a regular basis to detect changes in quality, resulting from any interventions introduced to the reporting processes, such as education programs [13]. The inter-rater reliability of AQUA-12 was excellent; it was robustly tested through ADRRC members from different specialties and also by the inclusion of three independent assessors external to the ADRRC review processes who are at different levels of seniority (two junior medical doctors and a clinical pharmacologist). Furthermore, AQUA-12 was designed to assess the practical application of knowledge and skills of reporting healthcare professionals through the completeness of information provided in ADR reports. Data elements in AQUA-12 are based on ADR principles and closely reflect the information required in the hospital ADR report form, which adheres to the reporting requirements by the Therapeutic Goods Administration, the national pharmacovigilance authority in Australia [14, 15].

Previous studies at our institution found that approximately 85% of ADRs were reported to ADRRC by hospital pharmacists, a rate similar to the current study [11]. Further, significant knowledge gaps exist among healthcare professionals regarding ADR principles that are important to assessment and management, particularly that of ADR syndrome recognition and causality attribution [14]. In keeping with these findings, the current study using the AQUA-12 tool also noted similar knowledge deficiencies, where reporters omitted detailed information regarding the actual reaction and description of key events around the ADR narrative (i.e. information relevant to syndrome recognition/diagnosis) and identification of suspected medications, which is important in causality attribution. In this study, we also attempted to identify if there might be specific drivers behind low-quality reports (as defined by AQUA-12 score of < 10) and found that there was a higher proportion of mild and fatal reactions, or reactions of unrecorded severity in reports that were deemed low quality. We postulate that attitudinal factors, such as diffidence and ignorance [16], may have contributed to the lack of effort in compiling a high-quality report, especially for mild or fatal reactions where perceived importance to subsequent patient management may have been diminished.

To address these knowledge gaps and attitudes, a multidisciplinary, multi-modular, interactive education program is currently being developed as a quality improvement initiative, to commence in late 2022. This education program will deliver the content in the following modules: (i) classification of ADRs, (ii) practical skills on recognition and diagnosis of common ADRs, (iii) basic immunological mechanisms and type B (allergic reactions) ADR pathogenesis, (iv) how to conduct a comprehensive causality assessment, (v) how to report an ADR (including professional responsibilities and attitudinal factors that influence reports) and (vi) providing risk communication to patients. One of the main applications of AQUA-12 will be to assess for any improvement in quality scores after each education module has been implemented.

The main limitation of this study is that the AQUA-12 tool was developed and evaluated at a single institution, hence, its generalisability may be limited. Nevertheless, the tool was developed specifically for quality improvement purposes within our institution, and it has been shown to be highly reliable for its intended function. Further to that, as the data elements of the AQUA-12 tool are based on key ADR principles and reporting requirements by the national pharmacovigilance authorities, a tool of similar nature could easily be adapted to suit the quality improvement initiatives at other institutions. Secondly, similar to a previous study, we observed preferential reporting of immunologically mediated ADRs (~ 80%) at our institution, while it is known that non-immunologically mediated reactions constitute a larger proportion of ADRs [17]. Further prospective evaluation over time is thus warranted to monitor the quality of reporting of non-immunologically mediated reactions.

Conclusion

This study demonstrated that AQUA-12 is a practical quality assessment tool that can be utilised in hospital settings to regularly monitor the completeness of ADR reports. Using AQUA-12, data elements decreasing the quality of ADR reports have been identified, which will guide quality improvement efforts through an ADR education program.