Defining clinical pharmacy and support activities indicators for hospital practice using a combined nominal and focus group technique

Background Although clinical pharmacy is a crucial part of hospital pharmacist’s day-to-day activity, its performance is not usually subject to a holistic assessment. Objective To define a set of relevant and measurable clinical pharmacy and support activities key performance indicators (cpKPI and saKPI, respectively). Setting Portuguese Hospital Pharmacies. Method After a comprehensive literature review focusing on the metrics already in use in other countries, several meetings with directors of hospital pharmacies were conducted to obtain their perspectives on hospital pharmacy practices and existing metrics. Finally, five rounds with a panel of 8 experts were performed to define the final set of KPIs, where experts were asked to score each indicator’ relevance and measurability, and encouraged to suggest new metrics. Main outcome measure The first Portuguese list of KPIs to assess pharmacists’ clinical and support activities performance and quality in hospital pharmacies. Results A total of 136 KPIs were assessed during this study, of which 57 were included in the original list and 79 were later added by the expert panel. By the end of the study, a total of 85 indicators were included in the final list, of which 40 are considered to be saKPI, 39 cpKPI and 6 neither. Conclusion A set of measurable KPIs was established to allow for benchmarking within and between Portuguese hospital Pharmacies and to elevate professional accountability and transparency. Future perspectives include the use of both cpKPIs and saKPIs on a national scale to identify the most efficient performances and areas of possible improvement.


Impacts on practice
• This work intends to promote the discussion around performance indicators and to raise awareness and know-how on the current and future role of the Hospital Pharmacies, by determining a framework to develop it assessment on a recurrent basis. • For the first time in Portugal, a set of relevant and measurable indicators are defined in order to assess hospitals' pharmacies performance, using a combined nominal group/focus group technique. • The definition of these performance indicators is considered to be a landmark in hospital pharmacy in Portugal and it is the first step to the first national study to assess all hospital pharmacies of the NHS. • Since an international benchmarking system is not established for hospital pharmacies, it is essential to cultivate an internal culture of activity monitoring and internal Introduction and impact of the services provided to patients is by quantifying and monitoring clinical activities through audits, service reviews, incident reports and surveys to patients, and by ensuring that complaint management and control procedures are in place [5,13,17] . A known strategy to track and continuously assess performance is through the use of clinical pharmacy Key Performance Indicators (cpKPIs) [6]. According to several studies, cpKPIs could be used to evaluate the quality of care [17][18][19], to help define a patients' healthcare expectations regarding a clinical pharmacist, to allow benchmarking within and between organizations, to elevate professional accountability and transparency [5], and to allow the tracking of the organization's progress towards achieving predefined goals and standards of care [4,5]. They also play an important part in rewarding good performance, in improving resource allocation and efficiency, and in identifying and reducing clinical errors, whilst maximizing healthcare outcomes and balancing patient's wants and needs [17][18][19][20]. Given the wide range of services provided, assessing pharmacists' productivity and quality of care is somewhat difficult [21,22]. Thus, it is also relevant to establish KPIs addressed to support activities (saKPI).
Despite the evidence supporting the importance of defining KPIs to quantify pharmacists contribution to patient care [5,19,20,[23][24][25][26], three main barriers were identified by several authors regarding their implementation: (i) resistance to change related to documenting clinical activities due to increased workload, practice environment constraints and competing priorities; (ii) disbelief of KPIs' real benefits, value and existing support from other pharmacists and hospital administrations; and, (iii) uncertainty of how to address quality versus quantity or the influence KPIs may have in the future of pharmacy practice [6,[27][28][29].

Aim of the study
In Portugal, most hospital pharmacies only collect some internal data for certification/accreditation purposes or adhoc situations. Currently, there is no national standard system for activity monitoring, nor any nationwide framework that enables comparisons/benchmarks amongst pharmacies' performances regarding their clinical or support activities. Thus, the main goal of this study is to define, for the first time in Portugal, a national set of relevant and measurable cpKPIs/saKPIs to assess the National Health System Hospital Pharmacies' performance and quality.

Ethics approval
As a service development and evaluation study, it was exempt from formal ethics approval. All study participants were given full information and provided signed, informed consent.

Study design
Consensus cpKPI/saKPI were determined using a combined nominal group/focus group technique, which combines the prioritisation process of a standard nominal group technique with the in-depth discussion of a focus group [47]. The expert panel was encouraged to assess both the relevance and measurability of the original candidate KPI, and to suggest new candidate indicators, for five rounds. After each round, an in-person panel meeting was held, promoting ingroup discussions about the candidate KPIs and to clarify questions regarding the definition of any new proposed KPI (Fig. 1).

Defining the KPIs
Two stages were developed to define the first list of cpKPI/ saKPI: (1) Literature review and exploratory meetings; (2) Expert panel rounds.

Round 1
In-person meeting Presentation of the project methodology Presentation of the original candidate KPIs to the expert panel

Round 2
In-person meeting Original candidate KPIs: presentation and discussion of the results from the assessment of both relevance and measurability

Round 3
In-person meeting

Round 4
In-person meeting New suggested candidate KPIs: presentation and discussion of the results from the assessment of both relevance and measurability  (1) Literature review and exploratory meetings Two investigators (HL and ARL) conducted a comprehensive literature review focusing on KPIs used in other countries. To account for existing national practices, meetings were held with several renown hospital pharmacists, who shared their perspectives on currently implemented practices and metrics. After this process, a list of 57 candidate KPIs was defined.
The annual European Association of Hospital Pharmacists (EAHP) survey is annually deployed to measure the progress, key barriers and drivers of the implementation of their six Statements: (i) Introductory Statements and Governance; (ii) Selection, Procurement and Distribution; (iii) Production and Compounding; (iv) Clinical pharmacy Services; (v) Patient Safety and Quality Assurance; and, (vi) Education and Research. Since several Portuguese hospital pharmacies already participate in the survey [1,48], each EAHP Statement was divided into several Assessment Areas and specific candidate KPIs were defined to assess each Area.
(2) Expert panel rounds To reach the final list of KPIs, five rounds with an expert panel were performed from January of 2019 to March of 2020.

Round 1
The first round presented to the expert panel the key project moments, the main goals and methodology, all of which previously defined with the Portuguese Pharmaceutical Society, and the 57 candidate KPIs, categorised by EAHP Statement.
After this round, the expert panel had four weeks to rate each indicator in two different dimensions: Relevance and Measurability. The former was defined as the ability to reflect the hospital pharmacy performance or the clinical pharmacist's direct impact on patient care, while the latter was defined as the ability to easily collect data to calculate the KPI within the hospital.
At the beginning of the study, three criteria were defined by the expert panel to determine which KPIs would be included in the final list: (i) if the average relevance score for each indicator was low (rating equal or lower than 3 points), the indicator would be excluded, regardless the measurability score; (ii) if the average relevance score was high (rating higher than 3 points) and the measurability low, the indicator would be excluded; and, (iii) if the average relevance and measurability scores were high, the indicator would be included in the final list.

Round 2
After collecting all panellist scores and calculating the average score for each indicator, a second in-person meeting was held to present and discuss the results regarding relevance and measurability of the original candidate KPIs. In each inperson meeting, consensus concerning indicators with scores close to cut-off points were obtained by majority.

Round 3
The expert panel was given four weeks to suggest new candidate KPIs per EAHP Statement. Following this time, a third in-person panellist meeting was held for discussion and clarification regarding the proposed KPIs and their definitions.

Round 4
The expert panel was then asked to rate the new set of suggested KPIs according to their relevance and measurability, using the five-point Likert Scale. The fourth in-person round took place after having all the scores calculated for each indicator.

Round 5: Final set of KPIs
Finally, after assessing both original and suggested candidate KPIs, a last in-person meeting was held to present the final list and to define the sources of information available within the hospitals to measure each KPI.

KPI Definition
Concerning the definition of the candidate KPIs, the authors agreed that all should: (i) reflect the current hospital pharmacists activities, (ii) be evidence-based, (iii) be aligned with clinical pharmacists' goals, objectives and practices, (iv) be feasible to measure, (v) be relevant to clinical outcomes, and (vi) be used across all types of Hospital Pharmacies (e.g., rural, urban, teaching or non-teaching hospitals). A glossary indicating each rational, measurement unit, targetpopulation and data-source was then prepared for each suggested candidate KPI.

The expert panel
A panel of eight experts was specifically selected for this project by the board members of the Portuguese Pharmaceutical Society, considering their professional curricula, expertise, and contributions for the development of clinical pharmacy in Portugal. These experts are renowned hospital pharmacists having also professional responsibilities since 1 3 they are Pharmaceutical Society representatives and members of pharmaceutical associations in Portugal.
As for their main characteristics, the average age was 48.7 years old, mostly females, with around 25 years of experience as a pharmacist and around 10 years of experience as hospital pharmacy director ( Table 1).
The definition of a panel consisting exclusively of pharmacists aims to ensure that the defined cpKPIs/saKPIs are unanimously agreed upon, and that they effectively measure their performance. In an analogy of the Gettysburg Address speech by former U.S. President Abraham Lincoln, this expert panel was created to define the cpKPI/saKPI list for Hospital Pharmacy "of the pharmacists, by the pharmacists, for the pharmacists".

Round 1 and 2: assessing the original list of the candidate KPIs
Following an extensive literature review, 57 candidate KPIs were included in the original list, categorized into six EAHP Statements (Table 2), where 22 were considered as saKPI, 33 cpKPI and 2 neither.
After assessing their relevance, only two were considered as "not relevant" (rating lower than 3 points) by the expert panel and therefore were excluded.
Regarding measurability, although 21 KPIs were considered relevant (rating equal or higher than 4 points), data collection capability was low. For example, the 'Number of pharmacists rounds' or the 'Number of adverse events reported by patients' were considered as some of the most relevant KPIs, however, the ability to measure them ranged from 1.5 to 2.1 points. Similarly, the three KPIs rated as 'Totally Impossible' to measure (rating equal or lower than 1.3 points) were also considered to be highly relevant (rating equal or higher than 4 points).

Rounds 3 and 4: expert panel suggested candidate KPIs
The following rounds sought not only to include the new suggested candidate KPIs (round 3), but also to assess their relevance and measurability (round 4) ( Table 3).
After the third round, the expert panel suggested 79 new KPIs: 37 saKPI, 38 cpKPI and 4 neither.
Concerning their relevance and measurability (round 4), six KPIs were considered as totally relevant (rating 5 points) by all panel members and easily measured (rating equal to or higher than 4 points).
After the fourth round, 26 of the total suggested KPIs were excluded, 5 due to their low relevance and 21 due to their low ability to be measured (rating equal or lower than 3.0 points).

Round 5: defining the final set of KPI
Finally, a last in-person meeting was held with the expert panel to present the final list of KPIs, and to define the sources of information to calculate each metric (round 5).
The expert panel defined a final list with 85 KPIs to assess all six EAHP Statements: 14 to assess the introductory statements; 10 on selection, procurement and distribution; 6 on production and compounding; 16 on clinical pharmacy services; 25 on patient safety and quality assurance; and 14 on education and research (Table 4). Concerning the type of KPI, 40 are saKPIs, 39 cpKPIs and 6 neither.

Discussion
Although being the first study to define a Portuguese set of saKPIs/cpKPIs to assess hospital pharmacy performance and quality based on EAHP Statement, other countries have deployed similar studies, as previously mentioned.
Though some authors argue that setting benchmarks by accreditation bodies or certifications by international organizations is the first step of healthcare quality cycle [49], only one study from Brazil referred to the importance of measuring the existence of updated written operational procedures for all clinical pharmacy activities [34]. About the KPIs included in the second statement, the 'Drugs stock turnover' comprises a commonly used criterion for assessing the efficiency of pharmacies' purchasing and supply chain [26]. Given the importance of this metric, three indicators were defined in the Magarinos-Torres et al.
(2007) study to assess the stock turnover, namely the number of medication units lost, the value spent in lost medication, and the existence of updated reports on medication availability [34].
Our expert panel included the highest number of KPIs in the "clinical pharmacy services" statement, which is aligned with the indicators referred across the literature. The KPI 'Existence of medication reconciliations up to 72 h after admission' is one of the most frequently mentioned in the literature [4,5,17,19,26,45]. Although the time interval defined for reconciliation varies between 24 to 72 h, there is a consensus across the literature that it is highly relevant and measurable, not only by identifying the existence or nonexistence of these reconciliations, but also by the importance of quantifying the proportion of patients who received formal documented medication reconciliation at discharge [4,26].
Aligned with several studies [4,19,26,45], KPIs related to patients' education and information sharing are highly recommended. In the Lloyd et al. (2017) study, the expert panel argues that patients have to receive written/verbal counselling before discharge, and that they should also receive a document with an accurate medication list detailing any therapy changes [26]. In two other studies, both panel groups defined a specific KPI to measure the proportion of patients who have face-to-face discussions about their medication before discharge [4,45].
Concerning the clinical pharmacy services, the development of outpatient pharmaceutical consultations was considered an important area of clinical intervention, aiming for medication reconciliation, drug interactions management, adverse reactions detection, patient education among others [50][51][52]. Therefore, outpatient pharmacy and consultation has become an important part of pharmacists' tasks [52,53].
As for the statement patient safety, although the number of adverse events reported by staff and/or by patients is one of the KPIs most frequently referred to in the literature, this indicator can assume different definitions. For example, in a study from Brazil, this KPI is mentioned as 'Number of problems that occurred related to medications' and 'Number of problems related with medications identified and notified' [34]; in a study from Belgium, it's referred to as 'Number of interventions accepted and activities performed to prevent, detect, assess, manage, report, and/or document adverse      [19]. Given that the Portuguese National Pharmacovigilance System requires pharmacists to report adverse events, the expert panel included a specific KPI to quantify the ratio between reported adverse events and number of discharges/ outpatients followed. Finally, all panel members agreed that pharmacist's continuous education was one of the key elements to ensure the quality of care and should therefore be assessed. Similarly, an Australian study concluded that continuous training is part of the learning process of clinical pharmacists in 239 public hospital pharmacy services [7]. Thus, it is not surprising that the cpKPI 'Time spent on training' was considered relevant by our expert panel, as well as by several studies [26,34,38]. According to this last study, participation in continuing long-term higher education usually includes areas such as expertise in ward pharmacy, medication reviews, or accreditation for comprehensive medication reviews [38].
The key outcome of this study was the definition of the first list of national KPIs to assess hospital's pharmacists' performance. Additionally, the study design enabled an important advantage, given the characteristics of the expert panel, which included individuals who feel more comfortable participating in face-to-face meetings rather than in multi-round surveys. The use of a combined nominal group/ focus group technique ensured equal participation amongst panellists and gave them the opportunity to explore diverging opinions throughout the group discussions. Finally, when implemented at the national level, we expect that these KPIs will improve transparency and accountability among hospital pharmacies and heighten the quality of care.
As for potential weaknesses, the exclusion of relevant KPIs by the expert panel based on the low measurability may limit the scope of assessment. However, since the indicators included in the final list reflect the dichotomy between high relevance and measurability, it is expected that its adoption by hospital pharmacies and by the Ministry of Health may become a reality before long.

Conclusion
Defining these 85 cpKPIs/saKPI is a first step towards assessing Hospital Pharmacy performance and quality. Major challenges are expected to arise during the implementation of these KPIs at a national level. Some of which include: defying the status quo, increasing workload in data collection, ensuring data quality and, most importantly, communicating across all players that KPI measurement to monitor performance in hospital pharmacists' clinical and support activities will allow better activity assessment, leading to an improvement in inpatient and outpatient quality of care, enabling continuous future development and planning with greater certainty.
Despite this study's major contribution to hospital pharmacists' clinical and support activities, future research should focus on gathering external stakeholders' feedback on relevant KPIs, developing consensus indicators for outpatient care or for subspecialty areas, which require different and/or supplemental metrics to help improve quality of patient care and further develop clinical pharmacy practice. Thus, future research ought to contribute to a more complete understanding of KPIs role in this field. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/.