FormalPara Key Points

For effective use of adverse drug reaction reports in pharmacovigilance, reports should contain sufficient information and apply feasible causality assessment tools.

The results of this study suggest that the new algorithm for specifically assessing fatal cerebral haemorrhage is more useful for assessing medicine causality using the information available in clinical practice (medical records) compared with existing methods.

1 Introduction

In pharmacovigilance, adverse drug reaction reports are an essential source of information. Causality assessment of individual cases forms the basis of safety measures for medicinal products [1, 2]. Since clinical trials involve only a limited number and characteristics of patients, information regarding suspected adverse drug reactions obtained from post-marketing clinical practice is crucial to ensure medicinal product safety [2,3,4]. Specifically, in adverse drug reaction reports with death as the outcome, the extent to which the drug affects the course of events leading to death should be carefully assessed when considering the need for prompt safety measures.

Reports of suspected drug reactions to regulatory authorities should contain sufficient information to be used effectively [2, 5]. Additionally, proper methods for assessing the causal relationship between drugs and adverse events should be established [6, 7].

Sufficient information on suspected drug reactions is a prerequisite for assessing the causal relationships between medicines and adverse events. Annually, the Japanese regulatory authority Pharmaceuticals and Medical Devices Agency (PMDA) receives approximately 60,000 reports on adverse drug reactions from medical practices and pharmaceutical companies [8]. However, most of these reports contain insufficient information [9]. Thus, clarifying the information to be included in a suspected adverse drug reaction report is necessary to enable causal relationship assessment and establish practical tools for this assessment [10, 11]. Such a method would allow for consensus among regulatory authorities, pharmaceutical companies, and medical institutions and thus facilitate consistent and efficient causality assessment.

Various methods have been developed for causality assessment [7], including the widely used Naranjo algorithm [12] and the World Health Organization-Uppsala Monitoring Centre (WHO-UMC) scale [13]. However, both methods lack specificity with respect to the content of the items since they are not event specific; instead, they are designed to generally deal with adverse effects. Furthermore, when analysing the relationship between medicinal products and adverse reactions, these methods include items that are challenging to observe in fatal cases, including adverse reactions disappearing after withdrawal (dechallenge) or reappearing after re-administration (rechallenge). Therefore, event-specific methods that help evaluate fatal cases are warranted.

Cerebral haemorrhage is an important and potentially life-threatening adverse reaction that is most frequently reported to the PMDA [9]. Cerebral haemorrhage is one of the most challenging events to assess for a causal relationship with a medicinal product owing to the diverse possible causes of its onset or exacerbation. In addition, 84% of fatal cerebral haemorrhage cases reported as suspected adverse drug reactions are published as ‘not assessable’ owing to a lack of information [9]. To overcome this, the first author previously proposed a novel algorithm (referred to here as ‘the ACAD-FCH: the causality assessment algorithm for fatal cerebral haemorrhage’ for convenience) that uses required information to assess the causal relationship between medicines and fatal cerebral haemorrhage [9].

The ACAD-FCH consists of two steps. The first step is ‘appropriate use of the drug’, ‘temporal association’, and ‘severity of brain haemorrhage (fatal or not)’; the second step is ‘assessment of the effect of the drug’ and ‘assessment of other factors’. The first step identifies the preconditions for assessing the causal relationship between the medicinal product and fatal brain haemorrhage, whereas the second step evaluates the causality balance between the suspected drug and other factors in fatal cerebral haemorrhage.

The use of the ACAD-FCH in clinical practice has not been verified. Thus, this study aims to explore the feasibility of implementing the ACAD-FCH in drug safety evaluation. Additionally, the assessment results from this algorithm were compared with those from the Naranjo algorithm and the WHO-UMC scale.

2 Methods

This exploratory study evaluated cerebral haemorrhage cases using information obtained from hospital electronic medical records to verify the feasibility of implementing the ACAD-FCH in drug safety evaluation. This was to determine whether the information required in the causality assessment using this algorithm was available in clinical practice. We also compared the assessment results from this algorithm with those from the Naranjo algorithm and the WHO-UMC scale.

2.1 Data Resource

The medical records of patients who died at the University of Tokyo Hospital between 1 January and 31 December 2020 and for whom a death certificate was created in the documentation system or death was registered as an outcome in the electronic medical records were reviewed. Cases with a head imaging scan [computed tomography (CT) or magnetic resonance imaging (MRI)], performed during the course leading to death and a documented intracranial haemorrhagic lesion, including a subarachnoid haemorrhage, subdural haematoma, and cerebral haemorrhage, were selected. Cases with unavailable medical records, without administered medication, or with unknown administration status before the haemorrhagic lesion were excluded.

2.2 Causality Assessment

Two board-certified neurosurgeons (MO and SM) independently reviewed the electronic medical records of all the included cases. They examined the association between the medicines used and the incidence of intracranial haemorrhage and death. Three tools were used to assess causality: the ACAD-FCH (Table 1), Naranjo algorithm (Table 2), and WHO-UMC scale (Table 3). The ACAD-FCH was used to assess the causal relationship between death due to intracranial haemorrhage and drug use. Contrastingly, the Naranjo algorithm and WHO-UMC scale were used to assess the causal relationship between drugs and intracranial haemorrhage since these tools lack items related to death. More information on these methods can be found in previous studies [9, 12, 13].

Table 1 The causality assessment algorithm for fatal cerebral haemorrhage (ACAD-FCH) [9]
Full size table
Table 2 Naranjo algorithm [12]
Full size table
Table 3 WHO-UMC scale [13]
Full size table

Regarding the ACAD-FCH and Naranjo algorithms, the numbers of ‘Yes’, ‘No’, and ‘No information/Do not know’ responses to each question in each algorithm by both evaluators were summed. The frequency of the ‘No information/Do not know’ response was compared using Fisher’s exact test. Statistical significance was set at P < 0.001.

2.3 Inter-Rater Reliability

Inter-rater reliability for each method was evaluated using percentages of agreement and kappa statistics with 95% confidence intervals (CIs). The kappa result was interpreted as follows: values < 0, none; 0.00–0.20, slight agreement; 0.21–0.40, fair; 0.41–0.60, moderate; 0.61–0.80, substantial; and 0.81–1.00, almost perfect agreement [14]. Computations were performed using STATA software (version 15.1 for Windows, STATA Corporation, College Station, USA).

3 Results

A total of 316 patients died at the University of Tokyo Hospital during the study period. Among them, 190 patients underwent head CT or MRI for diverse reasons, and 33 had intracranial haemorrhage. Of these, 24 patients who had received any medication before the haemorrhage were assessed. The Supplementary File contains details of the characteristics of the 24 cases, and the results of the assessments by the two evaluators.

Figure 1 illustrates the proportion of ‘Yes’, ‘No’, and ‘No information/Do not know’ responses selected for the nine and ten items of the ACAD-FCH and Naranjo algorithm, respectively, in the 24 case assessments recorded by both evaluators (i.e., 48 times for each question). The proportion of ‘No information/Do not know’ responses for each question, indicating insufficient information for a decision, was 35.6% (95% CI 31.4–40.6) for the overall ACAD-FCH and 66.9% (95% CI 62.5–71.1) for the overall Naranjo algorithm, with a significant difference (P < 0.0001). In the ACAD-FCH, the question with the least number of ‘No information’ selections was 3(i): ‘It was a serious cerebral haemorrhage’ (one response). Contrastingly, the question with the most frequent ‘No information’ selections was 2(ii): ‘Blood levels of the suspected drug were sufficiently high at the onset of cerebral haemorrhage’ (39 responses). In the Naranjo algorithm, questions 3, 4, 6, 8, and 9 were challenging or impossible to answer in assessing death cases, with the ‘Do not know’ response being selected in most cases.

Fig. 1
figure 1

The proportion of ‘Yes’, ‘No’, and ‘No information/Do not know’ responses selected by both evaluators for each question of the ACAD-FCH and Naranjo algorithm

Full size image

Table 4 presents the distribution of results (A–C) and inter-rater agreement (D) of the 24 cases assessed by both evaluators using the three methods. There were notable differences in the inter-rater agreement using the three evaluation methods. The ACAD-FCH had a high Cohen's kappa coefficient of 0.867, indicating substantial agreement beyond chance. Additionally, the agreement rate was 91.7%, with a narrow CI (95% CI 79.8–100). These findings suggest a strong consensus between both evaluators when using the ACAD-FCH.

Table 4 Causality assessment by two raters
Full size table

In contrast, the Naranjo algorithm yielded a considerably lower Cohen’s kappa coefficient of 0.139, implying a slight agreement beyond chance, with a relatively high agreement rate (70.8%). This discrepancy suggests that the true agreement level is limited when the Naranjo algorithm is used. The WHO-UMC scale yielded intermediate results with a Cohen’s kappa coefficient of 0.326, indicating a fair agreement. The agreement rate was 50.0%, with a relatively wide CI (95% CI 28.4–71.6). These outcomes suggest substantial variability in agreement between the observers.

4 Discussion

Our findings suggest that the event-specific ACAD-FCH is more useful for assessing medicine causality using the information available in clinical practice (medical records) compared with existing methods. The high frequency with which the evaluator was able to select ‛Yes’ or ‛No’ rather than ‛No information’ implies that the information required for evaluation using the ACAD-FCH is sufficiently available from the medical records, and thus it is feasible. In addition, the extremely high inter-rater agreement indicates that the ACAD-FCH is reliable.

These results may have been obtained because the ACAD-FCH excludes items, such as rechallenge, which cannot be easily determined when assessing death cases. The Naranjo algorithm and WHO-UMC scale include conclusive categories, such as ‘Definite’ and ‘Certain’, which require extensive solid evidence. However, removing the items with inaccessible data in death cases in both methods would not yield a conclusive result. The score range for the Naranjo algorithm is −4 to 13. However, after excluding items that are challenging to assess in critical cases, including death (question 3, 4, 6, 8, and 9), the score range becomes −2 to 7 points, and the assessment result cannot be ‘Definite’ (need ≥ 8 points). In the WHO-UMC scale, in cases in which it is challenging to determine the ‘response to withdrawal’, few cases are ‘Certain’ or ‘Probable’. Proving that the drug is 100% responsible is challenging, and this is not limited to fatal cases [6, 15]. Causality assessment can only be performed by determining whether the drug or other factors are more influential (causality balance) using the available information. The ACAD-FCH, which is structured mainly using items for which data can be easily obtained in clinical practice, such as temporal relationships, imaging findings, and patient background, allows for efficient assessment of the causality balance.

Another reason is that the ACAD-FCH has a two-step structure. Step 1 confirms the five items that form the premise for causal assessment. Step 2 assesses the balance of causal relationships with four items, which facilitates the correction of the assessment trajectory compared with those of the Naranjo algorithm or WHO-UMC scale, in which these items are assessed together. For example, in the ACAD-FCH, events without a temporal causal relationship are determined as ‘Unlikely’ in Step 1 and are not affected by the results of Step 2. In contrast, in the Naranjo algorithm, the score for no temporal relationship (−1 point) is cancelled out by the scores for the other items. Accordingly, the assessment result can be ‘Possible’ or indicate a stronger causal relationship [16]. In the WHO-UMC scale, the effect of the medication, other factors, and the temporal association are assessed on the basis of the evaluator’s impression. The assessment result is based on the one that fits all the criteria, leading to high variability between evaluators [17]. In the ACAD-FCH, nine items are rated in two steps; therefore, we could obtain inter-rater agreement beyond chance.

Previous studies using the Naranjo algorithm and WHO-UMC scale revealed agreement rates of 43–93% [18,19,20] and 24–73% [17, 20, 21], respectively, with kappa coefficients of 0.14–0.86 [18,19,20, 22] and 0.11–0.71 [17, 20, 21], respectively. The results vary on the basis of study conditions, including the subject of the evaluation. These values are consistent with our findings. Furthermore, the Naranjo algorithm is likely to have a ‘Probable’ or ‘Possible’ result [19]; and the WHO-UMC scale is not reproducible since it is based on individual judgement [17, 23]. Similar trends were observed in this study.

No single method is universally accepted [7]; nonetheless, the previously developed assessment methods have their advantages [20]. Moreover, using and comparing several assessment methods simultaneously or using different methods, depending on the situation, is beneficial. For example, a generic method is required when evaluating cases without specifying the event in which a specific drug was used. Contrastingly, when focussing on a specific event, an assessment method involving items appropriate to that event is more efficient. The ACAD-FCH is the only causality assessment method specific to drugs and fatal cerebral haemorrhage. Consequently, it could be useful when the analysis focusses on the occurrence of severe intracranial haemorrhage without identifying the suspected drug. This method is expected to contribute to compensating for the shortcomings of existing methods when assessing fatal cases with cerebral haemorrhage. Furthermore, while the ACAD-FCH is currently specific to cases of fatal cerebral haemorrhage, its applicability could extend to other side effects. This adaptation could be achieved by adjusting the details of the assessment items, or in cases where the outcome is other than death, by removing the item 3 (assessing whether the event was fatal). In cases where the involvement of multiple drugs is suspected, including instances where drug interactions may be the cause, the causality assessment can be conducted by replacing item 4 with the following: ‛Can the pathological condition be explained by the drug interactions?’

While the ACAD-FCH proves useful for extracting and organising the necessary information for causality assessment from large amounts of clinical information, its usefulness may diminish in instances where there is insufficient information, even in clinical practice. For instance, scenarios lacking a history of hospital visits and providing minimal background information could limit the algorithm’s utility. Additionally, if, for any reason, a head imaging scan cannot be performed, the usefulness of this algorithm might not be demonstrated, even in cases where cerebral haemorrhage is strongly suspected on the basis of clinical symptoms. From our perspective, information derived from head imaging studies plays a crucial role in assessing causality concerning drugs.

The ACAD-FCH was developed with the purpose of assessing individual cases, aiming to improve the quality of individual case reports and streamline the assessment process. However, to determine the true causal relationship between drugs and adverse events, rigorous large-scale controlled trials, controlling for confounding factors, become imperative. Numerous confounding elements, such as age and underlying medical conditions, can be associated with head imaging tests and drug use. This complexity makes it challenging to assess the causal relationship between drugs and cerebral haemorrhage. The ACAD-FCH has the potential to facilitate early signal detection, potentially prompting controlled trials and creating a virtuous circle wherein trial outcomes contribute to refining the assessment of the causality balance.

This study has several limitations. First, both evaluators are neurosurgeons; therefore, their judgement may be similar. Therefore, future studies should consider assessments by doctors from other medical specialities or pharmacists. Second, since this study was based on data from a single centre, it is difficult to extrapolate the results, and further multi-centre studies are warranted. In addition, this study did not analyse intra-rater agreement, which should be performed after a certain period in the future. Furthermore, given the insufficient number of cases for comparing drug- and non-drug-related cerebral haemorrhages and the potential bias introduced by the COVID-19 pandemic in 2020, an assessment of cases from the pre-pandemic period becomes imperative—a challenge to be addressed in future. Overcoming these challenges and obtaining additional data from diverse real-world sources will enable the validation of the usefulness of ACAD-FCH by analysing its sensitivity, specificity and positive predictive value in comparison with existing databases. Once the algorithm’s robustness is confirmed, there may be an opportunity to develop it into a compelling study comparing artificial intelligence (AI) and human assessments.

5 Conclusion

This study explored the usefulness of the ACAD-FCH, a method developed to assess the causal relationship between drugs and fatal cerebral haemorrhage, using medical information obtained from cases of intracranial haemorrhage where death occurred. Compared with the Naranjo algorithm and WHO-UMC scale, which are widely used, the results suggest that the disease-specific ACAD-FCH may be more effective in assessing intracranial haemorrhage cases with death. However, larger studies, including intra-rater assessments, are warranted to validate this algorithm.