Risk prioritisation algorithms provide patients with a risk category that guides pharmacists to choose those needing medication reviews (MRs) the most. For this study the Medicine Risk Score (MERIS) and a modified Assessment of Risk Tool (ART) were used.
To examine how the selection of patients by the clinical pharmacists in an emergency department for MRs compared with the categorisation provided by MERIS and a modified version of ART (mART). Furthermore, examine the agreement between MERIS and mART.
A cross-sectional study was conducted using data on all admitted patients during a two-month period. Data were entered into the prioritisation algorithms and independently ranked by the six pharmacists who were observed as they selected patients for MR. Risk scores and categorisations were compared between the algorithms and the pharmacists’ ranking using t-test, Z-test, Chi square, Kruskal Wallis H-test, or Kappa statistics.
The study included 1133 patients. Significant differences were found between the pharmacists and the algorithms. The sensitivity and specificity of MERIS were 37.8% and 73.6%, for mART, 33.0% and 75.9%. Kappa was 0.58, showing moderate agreement. No significant differences were observed between the individual pharmacists’ selection, but differences were significant between how pharmacists ranked the importance of the provided MRs.
Pharmacists disagreed with the risk categorisation provided by MERIS and mART. However, MERIS and mART had similar sensitivity, specificity, and moderate agreement. Further research should focus on how clinical algorithms affect the selection of patients and on the importance of the MRs carried out by pharmacists.
Similar content being viewed by others
Non-clinical ground rules disagree with risk prioritisation algorithms that are based on clinically important risk factors.
This comparison can be used to provide pharmacists with a better foundation on which to decide between risk prioritisation algorithms in a clinical setting.
Medicine Risk Score and a modified version of Assessment of Risk Tool had a moderate agreement in their risk categorisation.
Medication reviews (MRs) aim to improve the quality, safety, and appropriate use of medications by finding and solving drug-related problems (DRP) [1,2,3,4]. Important risk factors for DRPs include polypharmacy, female gender, high risk medications, renally eliminated drugs and being elderly (most often 65 years of age and older) [2, 3]. Some of these risk factors are also used as criteria for patient prioritisation for MRs [5, 6].
At the joint emergency department (JED) at Nordsjællands Hospital in Denmark, pharmacists provide medication reviews (MRs) to hospitalised patients. Risk prioritisation algorithms could help find patients for whom the pharmacist's resources are best utilised [7, 8]. Several risk prioritisation algorithms have been developed, such as the Brighton Adverse Drug Reactions Risk model and the Medicines Optimisation Assessment Tool. These risk prioritisation algorithms include different items used to calculate the risk of DRP or measure other similar outcomes [9,10,11,12,13,14,15,16,17,18,19,20].
The JED at Nordsjællands Hospital has a high patient flow and only one pharmacist providing MRs on any given day, so it was essential to select the patients that would benefit the most from MRs [7, 21]. Currently, pharmacists at JED do not use a prioritisation algorithm but use non-clinical ground rules developed by JED. The non-clinical ground rules prioritise patients with the longest stay and who have not had their medications transferred to the electronic health record (EHR) . Therefore, it was important to examine the current non-clinical ground rules and compare current practice with the risk categorisation of the prioritisation algorithms.
This study used the algorithm Medicine Risk Score (MERIS) developed in Denmark using literature search and the Delphi method and a modified version of Assessment of Risk Tool (ART) developed in New Zealand using a group consensus method. MERIS consists of three items: reduced renal function, number of drugs and number of drugs with different categories of risk of harm and risk of interaction. ART consists of 38 items divided into five categories: patient profile, patient encounter, clinical profile—patients with chronic diseases, high-risk medications and laboratory values. Both algorithms provide risk categories by adding points from the fulfilled items [23,24,25,26].
To examine how the selection of patients by the clinical pharmacists in an emergency department for MRs compared with the categorisation provided by MERIS and a modified version of ART (mART). Furthermore, examine the agreement between MERIS and mART.
No ethical approval was required according to Danish law , however the pharmacists were anonymous and all material was stored confidentially. Data collection and processing were in compliance with the European General Data Protection.
A cross-sectional study was conducted at the JED at Nordsjællands Hospital in Denmark. Patient data relevant for the algorithms were collected while observing the pharmacists working at the JED, who provided MRs daily from 8 a.m. to 5 p.m. All admitted patients 18 years of age and older were included the study, as well as patients who stayed overnight and patients admitted until approximately 4:45 p.m. Afterwards, patient data were entered into the risk prioritisation algorithms MERIS and the modified ART (mART). When ART was developed, it had 38 items . However, some of these items were specific to the New Zealand health care setting or not suitable for a fully automatic algorithm, or relevant in the JED. Therefore, 12 items were removed to make the mART used in this study. Examples of items removed were: being from specific New Zealand native populations, poor English proficiency, history of psychiatric disease, and having specific laboratory values taken over several days. See electronic supplementary material 1 for the mART items.
The first author (SGK), a student in her last semester prior to graduating as a pharmacist, followed six pharmacists working at JED. Each pharmacist was observed for three days from March 9th to May 6th, 2022, on weekdays and weekends. The pharmacists ranked their MRs as ‘not important’, ‘important’ or ‘very important’, based on their clinical insight. The pharmacists did not change their typical workday during the data collection, meaning that they continued to attend meetings and precept students. These days, they did not manage to prioritize and complete as many MRs as they might have otherwise. The overall study included both the patients who were actively deselected for MRs and those admitted patients whom the pharmacists did not have time to consider.
The algorithms Medicine Risk Score (MERIS) and a modified version of Assessment of Risk Tool (mART) were used. Besides the relevant information for the different items in MERIS and mART, information about the patients’ age and gender were collected. The patients’ medication lists were extracted manually from the EHR and entered into an Excel spreadsheet. All medication formulations were counted (i.e. tablets, cremes, eyedrops). Duplicate medications and combination medications were counted once per active ingredient. The medication lists were taken from the EHR and not from patient interviews. Unless otherwise stated it was assumed that every drug on the list was used by the patient.
A frequency analysis of the total mART risk scores made the cut-off values for the risk categories of the mART. As observed in the original ART development the 10%, 15%, and 75% cut-offs were used . The mART cut-off values were based on a frequency analysis of the total ART risk scores. Patients with a risk score above 36 were defined as high-risk, 25–36 as medium-risk, and 24 or under were low-risk. In the original version of ART the cut-off values were: above 22 were defined as high-risk, 11–22 as medium-risk, and 10 or under were low-risk .
The mean ages of patients in the high versus those in the low-risk categories of the algorithms were compared as were mean risk scores of patients who had either been selected or deselected by the pharmacists. Proportions of females were compared in the algorithms' high versus low-risk categories. Finally, the pharmacists’ ratings of the importance of the MRs were compared to the risk categorisations provided by the algorithms.
The sensitivity and specificity of the MERIS and mART were calculated using the pharmacists’ selection and deselection as the gold standard. For mART the medium and high-risk categories were merged. A true positive and a true negative were when the algorithms and the pharmacists agreed and a false positive and a false negative when they disagreed. The sensitivity, specificity and Kappa were calculated twice, respectively including and excluding patients with missing laboratory values.
The pharmacists’ numbers of patients selected and deselected for MRs were compared among them and their risk categories against MERIS and mART. In addition, the ratings of importance were compared among the individual pharmacists.
T-tests were used for comparing means, Z-tests for proportions. The Kappa Statistics test assessed the agreement between the algorithms’ risk categorisation. Finally, Kruskal–Wallis H, Chi square and Monte Carlo significance tests were used for nominal variables. All statistical analyses were done in SPSS version 126.96.36.199.
In total, 1133 patients were included in the analysis of MERIS and mART, where pharmacists actively selected or deselected from 450 patients. Approximately 70% of the patients were categorised as low-risk on both the MERIS and mART tools (Table 1).
Statistically significant differences were observed in the mean age between the risk categories for MERIS and mART, where patients classified as high-risk had a higher mean age. Similarly, patients selected by pharmacists had a higher mean age than those deselected for MRs. No statistically significant gender differences were observed with this categorisation (Table 2).
No statistically significant differences were found between the risk scores of selected and deselected patients for MRs in the high-risk category of MERIS, indicating that the risk scores were similar. This also applied to the high-risk and medium-risk categories of mART. However, for the low-risk categories of MERIS and mART, statistically significant differences were seen between patients who were selected and deselected by pharmacists, where the highest risk score was observed in the selected patients (Table 3).
A statistically significant difference was observed between the pharmacists’ rating of importance of providing MR to the patient and the categorization of patients by both algorithms. However, for both algorithms there was a larger number of ‘not important’ ratings by the pharmacists in the low-risk categories, indicating that there might be some agreement between the pharmacists and low-risk categorisation by the algorithms (Table 4).
The sensitivity and specificity for MERIS were 37.8% and 73.6%, respectively when the patients with missing laboratory values were included. When these patients were removed the values were 37.9% and 72.0%. The sensitivity and specificity for mART were 33.0% and 75.9% respectively, and 33.3% and 74.0% when removing patients with missing laboratory values. The agreement of risk categorisation between MERIS and mART (grouping the high-risk and medium-risk groups of mART) was statistically significant (Kappa = 0.58), indicating a moderate agreement . The same applied to patients when excluding patients with missing laboratory values (Table 5).
When looking at the items of mART, three were not fulfilled by any of the patients. These were the following: a positive Clostridium difficile toxin culture, a therapeutic drug monitoring test, and an activated partial thromboplastin time of over 100 s. Generally, the items relating to laboratory were fulfilled by a low proportion of patients. The estimated glomerular filtration rate (eGFR) < 30 ml/min and P-glucose > 11 mmol/L were the items most frequently used (5.7% and 3.6% of all patients). The most frequently fulfilled mART items were: having more than 8 drugs and being over 75 years old (approximately 40% of the patients). In MERIS every item was fulfilled by at least one patient. The most frequently fulfilled items were drugs with low and medium risk of interaction and drugs with medium risk of harm (96.9% and 82.1%).
No statistically significant difference was observed between the individual pharmacists’ number of selected and deselected patients. However, Pharmacist 3 stood out as selecting the most patients for MRs and deselecting the fewest patients. A statistically significant difference between the individual pharmacists’ rating of importance of the provided MRs was observed. Pharmacist 1 had the largest proportion of ‘very important’ MRs, whereas Pharmacist 3 had the largest proportion of ‘not important’ MRs. No statistically significant difference between the individual pharmacist’s number of MRs in each risk category of the two algorithms. For most of the pharmacists there was a larger proportion of MRs in the low-risk category of both algorithms (Table 6).
Statement of key findings and interpretation
The cut-off values of mART corresponded to the distribution of patients in MERIS. Since mART had fewer items than the original ART and the maximum score was lower, naturally the risk category cut-off values changed. The mART cut-off values were higher than those of the original algorithm, which could be due to this patient population having higher risk scores than the population used in the development of original ART.
A higher mean age in the high-risk categories is not surprising as high age is an item in mART. Higher age is associated with using more medications, which could explain why those in the high-risk category of MERIS also had a higher mean age, as MERIS includes number of drugs as an item, but not age . The same difference was observed in the pharmacists’ selection, indicating that they may believe that older patients are more likely to have a DRP. No gender differences were observed in the risk categories, despite other studies showing that more women experienced polypharmacy [29,30,31].
Patients selected or deselected by pharmacists were at a similar risk of a DRP according to MERIS and mART. This indicates that the pharmacists disagreed with the risk categorisation provided by the algorithms. The non-clinical ground rules could also have influenced the selection and deselection, as the rules did not consider clinical risk factors . It could also be a coincidence that the deselected patient had high risk scores since they were deselected for various reasons. However, in the low-risk categories of MERIS and ART significant differences were observed between the selected and deselected patients, where the selected had a higher risk score. This indicates that the pharmacists selected patients at higher risk within the low-risk category. The study also showed differences between the pharmacists’ rating of importance and the risk categorisation provided by the algorithms, indicating that they disagreed with the risk categorisation.
The sensitivity and specificity of MERIS and mART were similar, indicating an agreement in the risk categorisation by the algorithms. Previous studies found that the sensitivity and specificity of MERIS were 64 to 81% and 52 to 75% [23, 32]. The specificity of MERIS for this study was similar. However, the sensitivity was lower, which could be due to using the pharmacists as the gold standard as their non-clinical ground rules did not necessarily take the risk of DRP into account. No sensitivity and specificity values have been reported earlier for mART. However, the same issue of the pharmacists’ non-clinical ground rules could also influence the relatively low sensitivity found in this study. A moderate agreement of risk categorisation by MERIS and mART was found. With the distribution of patients across the risk categories, it seems probable that the algorithms would agree since approximately 30% were at high-risk, and 70% were at low-risk.
The laboratory items of the mART were the items most often missing from the EHR. It could be due to the study taking place at the JED, where the patients did not stay for long and few had blood work done. Including many laboratory items is probably not applicable in this setting since mART was not explicitly developed for emergency departments . The most fulfilled items of mART match those from its validation study . All the items of MERIS were fulfilled and easy to collect from the EHR compared to mART.
Differences were found between the individual pharmacists in the number of patients they each selected or deselected for MRs. Research indicates an association between clinical experience and how pharmacists prioritise patients, where the more experienced often take the reason for admission into account . Clinical experience did not seem to be a factor in how pharmacists prioritized patients in this study, although pharmacists with the least clinical experience rated the largest proportion of patients as ‘very important’, indicating that they may generally feel more responsibility for providing MRs. All pharmacists rated a relatively large proportion of patients as ‘not important’, which again could be due to their non-clinical ground rules. Differences between pharmacists could be because they made decisions at different speeds or some did not necessarily follow the non-clinical ground rules, or due to alternating workdays with administrative or education tasks.
No difference was observed between the individual pharmacists’ number of MRs in the different risk categories of the algorithms. However, all the pharmacists had a larger number of MRs for patients in the low-risk categories, which could be due to their adhering to the non-clinical ground rules. This could also be due to the algorithms' categorisation of approximately 70% of patients in the low-risk category, which could explain why a higher proportion of low-risk patients were selected by pharmacists.
Strengths and limitations
One of the study’s strengths was that the data were collected on three alternate days for each pharmacist to get a larger data set with fewer day-to-day variations. To our knowledge a comparison of MERIS and ART has not been done before on the same patient population.
A limitation of the study was that the medication lists were based on the EHR, which was not always correct or updated. A recent Danish study has shown that 81% of the patients included had at least one discrepancy between EHR and what the patients reported taking at home . If this applied to the patients in this study, it could bias the risk categorisation. This could have been avoided by basing the medication lists on patient interviews. Also, the pharmacists did not rate the same patients, so it was not possible to conclude strongly about their similarities or differences. An independent expert could have rated the patients for a more objective comparison. However, since the pharmacists were supposed to work according to the same non-clinical ground rules and provide a similar performance, it seemed reasonable to put them in the same group to get a more realistic picture. Conducting this study only in one ED is a limitation and needs to be considered if these results are transferred.
A further limitation was a potential observer effect, as the pharmacists knew that the observer was noting the number of patients selected and deselected for MRs. This could have influenced how the pharmacists worked by making them work faster and selecting a higher number of patients than they normally would. The observation could also have influenced which patients the pharmacists selected, although it was not possible to know in which way this would affect the results.
Future research should have the pharmacists use one of these risk algorithms to examine how it affects their prioritisation and the importance of the MRs they then provide. It could be interesting to further characterize the patients where the algorithms disagreed in the risk categorisation and delve into the disagreements between the pharmacists’ categorizations and risk algorithms. Further work should also examine if there could be an association between the risk categorisation by the algorithms and the reason for admission.
The study found that the pharmacists disagreed with the risk categorisation provided by MERIS and mART. This could be due to the pharmacists' non-clinical ground rules, which did not consider clinical risk factors when selecting patients for MRs. However, MERIS and mART were found to have similar sensitivity, specificity, and moderate agreement in their risk categorisation. Further research should focus on how clinical algorithms affect the selection of patients and the importance of the MRs carried out by pharmacists.
Blenkinsopp A, Bond C, Raynor DK. Medication reviews. Br J Clin Pharmacol. 2012;74(4):573–80.
Krähenbühl-Melcher A, Schlienger R, Lampert M, et al. Drug-related problems in hospitals a review of the recent literature. Drug Saf. 2007;30(5):379–407.
van den Bemt PM, Egberts TC, de Jong-Van Den Berg LT, et al. Drug-related problems in hospitalised patients. Drug Saf. 2000;22:321–33.
Hellebek A, Marinakis C. Patientsikkerhed og lægemidler. Available from: https://pro.medicin.dk/specielleemner/emner/550. Accessed 01 Jun 2022.
Botelho SF, Pantuzza LLN, Marinho CP, et al. Consensus on the criteria for patient prioritization in hospital clinical pharmacy services: a Delphi study. Int J Clin Pharm. 2022;44(4):985–92.
Linkens AEMJH, Janssen MJM, van Nie N, et al. Additional value of a triggerlist as selection criterion in identifying patients at high risk of medication-related hospital admission: a retrospective cohort study. Int J Clin Pharm. 2022;44(5):1205–10.
Fernandes O, Shojania KG. Medication reconciliation in the hospital: what, why, where, when, who and how. Healthc Q. 2012;15:42–9.
Lewis P. Right patient, right time, right pharmacist: the time for clinical prioritisation tools? Eur J Hosp Pharm. 2017;24:314.
Sakuma M, Bates DW, Morimoto T. Clinical prediction rule to identify high-risk inpatients for adverse drug events: the JADE Study. Pharmacoepidemiol Drug Saf. 2012;21(11):1221–6.
Crutzen S, Schuling J, Hugtenburg JG, et al. Development and piloting of an algorithm to select older patients for different types of medication review. Front Pharmacol. 2019;10:217.
Kiguba R, Karamagi C, Bird SM. Incidence, risk factors and risk prediction of hospital-acquired suspected adverse drug reactions: a prospective cohort of Ugandan inpatients. BMJ Open. 2017;7:10568.
Alshakrah MA, Steinke DT, Tully MP, et al. Development of the adult complexity tool for pharmaceutical care (ACTPC) in hospital: a modified Delphi study. Res Social Adm Pharm. 2021;17(11):1907–22.
Hohl CM, Partovi N, Ghement I, et al. Impact of early in-hospital medication review by clinical pharmacists on health services utilization. PLoS ONE. 2017;12(2):e0170495.
Hohl CM, Yu E, Hunte GS, et al. Clinical decision rules to improve the detection of adverse drug events in emergency department patients. Acad Emerg Med. 2012;19:640–9.
de Winter S, Vanbrabant P, Laeremans P, et al. Developing a decision rule to optimise clinical pharmacist resources for medication reconciliation in the emergency department. Emerg Med J. 2017;34(8):502–8.
Pippins JR, Gandhi TK, Hamann C, et al. Classifying and predicting errors of inpatient medication reconciliation. J Gen Intern Med. 2008;23(9):1414–36.
Urbina O, Ferrández O, Grau S, et al. Design of a score to identify hospitalized patients at risk of drug-related problems. Pharmacoepidemiol Drug Saf. 2014;23(9):923–32.
Geeson C, Wei L, Franklin BD. Development and performance evaluation of the Medicines Optimisation Assessment Tool (MOAT): a prognostic model to target hospital pharmacists’ input to prevent medication-related problems. BMJ Qual Saf. 2019;28:645–56.
Geeson C, Wei L, Franklin BD. Medicines Optimisation Assessment Tool (MOAT): a prognostic model to target hospital pharmacists’ input to improve patient outcomes Protocol for an observational study. BMJ Open. 2017;7(6):e017509.
Tangiisuran B, Scutt G, Stevenson J, et al. Development and validation of a risk model for predicting adverse drug reactions in older people during hospital stay: brighton adverse drug reactions risk (BADRI) model. PLoS ONE. 2014;9(10):e111254.
Akutmodtagelsen – Hillerød. Available from: https://www.nordsjaellandshospital.dk/afdelinger-og-klinikker/akutafdelingen/akutmodtagelse-og-akutklinikker/Sider/akutmodtagelse-hilleroed.aspx. Accessed 24 Feb 2022.
Almarsdóttir AB, Haq R, Nørgaard JDSV. Prioritizing patients for medication review by acute ward pharmacists a mixed methods study in a Danish hospital. Int J Clin Pharm. 2022;5:1–10. https://doi.org/10.1007/s11096-022-01515-3.
Saedder EA, Lisby M, Nielsen LP, et al. Detection of patients at high risk of medication errors: development and validation of an algorithm. Basic Clin Pharmacol Toxicol. 2016;118(2):143–9.
Bonnerup DK, Lisby M, Sædder EA, et al. Effects of stratified medication review in high-risk patients at admission to hospital: a randomised controlled trial. Ther Adv Drug Saf. 2020;20(11):2042098620957142.
Falconer N, Nand S, Liow D, et al. Development of an electronic patient prioritization tool for clinical pharmacist interventions. Am J Health Syst Pharm. 2014;71(4):311–20.
Falconer N, Liow D, Zeng I, et al. Validation of the assessment of risk tool: patient prioritisation technology for clinical pharmacist interventions. Eur J Hosp Pharm. 2017;24(6):320.
Law LBK nr 1083 af 15/09/2017 §14 stk.2. Available from: https://www.retsinformation.dk/eli/lta/2017/1083. Accessed 31 Aug 2022.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74.
Kornholt J, Christensen MB. Prevalence of polypharmacy in Denmark. Dan Med J. 2020;67(6):A12190680.
Page AT, Falster MO, Litchfield M, et al. Polypharmacy among older Australians, 2006–2017: a population-based study. Med J Aust. 2019;211(2):71–5.
Loikas D, Wettermark B, von Euler M, et al. Differences in drug utilisation between men and women: a cross-sectional analysis of all dispensed drugs in Sweden. BMJ Open. 2013;3(5):e002378.
Høj K, Pedersen HS, Lundberg ASB, et al. External validation of the Medication Risk Score in polypharmacy patients in general practice: a tool for prioritizing patients at greatest risk of potential drug-related problems. Basic Clin Pharmacol Toxicol. 2021;129(4):319–31.
Andersen TS, Gemmer MN, Sejberg HRC, et al. Medicines reconciliation in the emergency department: important prescribing discrepancies between the shared medication record and patients’ actual use of medication. J Pharm. 2022;15(2):142.
We want to thank the pharmacists from the hospital pharmacy at Nordsjællands Hospital who participated in this study.
Open access funding provided by Royal Danish Library. No funding was obtained for this study.
Conflicts of interest
One of the authors (LM) was employed as clinical pharmacist at Nordsjællands Hospital at the time of the study and practiced at the ED. The other authors declare that they have no conflict of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Below is the link to the electronic supplementary material.
About this article
Cite this article
Korup, S.G., Almarsdóttir, A.B. & Magnussen, L. Comparison of prioritisation algorithms for the selection of patients for medication reviews in the emergency department: a cross-sectional study. Int J Clin Pharm 45, 884–892 (2023). https://doi.org/10.1007/s11096-023-01582-0