Introduction

Background

Residents living in Residential Aged Care Facilities (RACFs) are at high risk of medication-related problems [1] which can lead to medication-related harm. Medication-related harm is the overarching term used to describe harm amongst patients caused by medication errors and unsafe medication practices ranging from prescribing of potentially inappropriate medication through to dispensing and administration errors [2]. Internationally, inappropriate medication use impacts around 50% of residents living in RACFs [3] and it has been suggested that 95% of residents living in Australian RACFs have at least one-medication related problem [4]. Residents living in RACFs are also more likely to be prescribed potentially inappropriate medications compared to older people living at home [5,6,7] which may increase their risk of experiencing medication-related harm.

The importance of improving medication managementFootnote 1 processes to potentially help reduce medication-related harm is illustrated by the inclusion of medication management – polypharmacy and medication management – antipsychotic as quality indicators by the Aged Care Quality and Safety Commission [8]. This means that medication management is recognised as an important quality of care aspect which has the potential to impact upon the health and wellbeing of residents living in Australian RACFs [8]. There have been ongoing efforts to improve medication management within Australian RACFs. A 2017 pilot study conducted in Canberra, Australian Capital Territory (ACT) identified some promising findings associated with having an on-site pharmacist (OSP) working in a RACF [9]. Following on from this pilot study, in 2020, an OSP intervention was implemented as part of a cluster randomised controlled trial (RCT) in RACFs in Canberra, ACT [10]. The Pharmacists in Residential Aged Care Facilities (PiRACF) study evaluated the effectiveness and implementation of a 12-month OSP intervention which sought to improve medication management.

Implementation fidelity is commonly described as the extent to which an intervention was delivered as intended [11]. Implementation fidelity may be considered a core process evaluation component [12] or an aspect of implementation [13]. Historically, implementation fidelity has been seldom assessed when evaluating pharmacist interventions in health care settings [13, 14] but has begun to change recently [15,16,17,18,19]. Assessment of implementation fidelity can help inform whether the intervention’s outcome was due to design issues (i.e. theory failure) or intervention delivery issues (i.e. implementation failure), thus supporting real-time intervention delivery modifications and adoption of the intervention [20].

This mixed methods study was conducted within the context of the PiRACF study to understand the extent to which the OSP intervention was delivered as intended and determine the factors which influenced this intervention delivery across the seven intervention RACFs. This increased understanding has the potential to optimise the roll out of OSPs in Australian RACFs by determining whether the OSP intervention can be delivered as intended in real world RACFs. The identification of potential barriers and facilitators to successful OSP intervention delivery is also timely given that the Australian Government will be funding OSPs in RACFs from 2023 [21].

Aim

The aim of this study was to evaluate the implementation fidelity of the OSP intervention and understand the moderating factors which may have impacted delivery of the OSP intervention.

Methods

Study design

This study’s focus on intervention delivery is consistent with Gearing et al.’s assertion that intervention delivery is a core aspect of implementation fidelity [22]. The use of mixed methods study design for this study is consistent with previous health care implementation fidelity studies [17,18,19, 23,24,25,26]. A convergent parallel design was employed in this study wherein the quantitative and qualitative data were collected and analysed separately and then merged and integrated [27, 28]. This approach helps to offset any potential weaknesses associated with the individual data sets. The use of an existing implementation fidelity framework is also consistent with previous health care implementation fidelity studies [12, 17, 24, 29].

Hasson’s conceptual framework for implementation fidelity

Implementation consists of adherence components (measurable) and moderating factors (non-measurable) which inform and can influence fidelity [11]. For this study, Hasson’s conceptual framework for implementation fidelity was used as it expands upon Carroll et al.’s seminal conceptual framework for implementation fidelity [11] by proposing the inclusion of context and recruitment as additional moderating factors [30]. Please see Table 1 for further definitions of adherence and moderating factors within the implementation fidelity context. Hasson’s conceptual framework has also been previously employed to assess the implementation fidelity of a pharmacist intervention [17].

Table 1 Implementation fidelity terms and definitions adapted from Carroll et al. [11] and Hasson [30]
Full size table

Adherence assessment

In this study, implementation fidelity adherence was assessed based upon quantitative data, consistent with previous health care implementation fidelity studies [12, 19, 25, 29]. The three quantitative data sets selected for this study related to: (1) range of OSP intervention activities delivered; (2) random sample of 10% of medication reviews assessed for quality; (3) proportion of residents who received at least one medication review. These quantitative data sets were chosen given the pragmatic nature of this study and as they provided insights into the extent of both resident and RACF level activities delivered as part of the OSP intervention.

Moderating factors

The moderating factors influencing OSP intervention delivery were identified based upon qualitative data, consistent with previous health care implementation fidelity studies [12, 18, 24, 25, 29]. Explicit consideration of the moderating factors in this study is consistent with Bragstad et al.’s suggestion that adherence results need to be contextualised, thereby facilitating a more holistic understanding of implementation fidelity [12].

Participants and data collection

Activities undertaken as part of OSP intervention delivery were reported by OSPs via online pharmacist diaries on the Qualtrics survey platform [31]. These activities were within the current scope of practice of pharmacists registered with the Australian Health Practitioner Regulation Agency and were categorised into the following nine activities: clinical audits, medication reviews, communication, administrative tasks, vaccination, education, quality improvement and other (including fire safety training, signing statutory declarations and other facility level activities). OSPs were also asked to maintain a written copy of medication reviews undertaken for residents.

Managers and OSPs across each of the seven intervention RACFs were invited by email to participate in semi-structured interviews with a purposive (stratified) sampling approach employed [32]. These two participant groups were selected as it was considered that they would have the most insight into OSP intervention delivery and the factors influencing delivery of this intervention. The RACF manager and OSP interview guides used in this study have been published elsewhere [33]. Interviews were audio-recorded, transcribed and deidentified to help maintain participant anonymity and confidentiality [34].

Data analysis

The online pharmacist diaries were downloaded from Qualtrics and cleaned and checked in Microsoft Excel. The proportion of residents who received at least one medication review was determined by the number of written medication reviews provided by OSPs to the study team. A random sample of 10% of these written medication reviews were assessed for quality by two Medication Management Review Accredited Pharmacists (accredited pharmacistsFootnote 2) using a checklist adapted from Curtain’s medication review quality assessment work [35]. The medication review quality assessment approach taken in this study builds upon the Care Home Independent Prescribing Pharmacist Study (CHIPPS) wherein a random sample of pharmaceutical care plans were reviewed for appropriateness by suitable experts [36]. For this study, the Intraclass Correlation Coefficient between the two accredited pharmacists was also assessed. This approach is consistent with Gearing et al.’s recommendations regarding the use of two or more independent raters and assessment of inter-rater reliability [22]. For each of the three quantitative data sets, the study team developed an adherence scoring system, as well as an overall implementation fidelity adherence scoring system consistent with Bragstad et al.’s implementation fidelity rating system of low, medium or high [12]. Please see Additional file 1 for further details of these scoring systems. The use of these objective adherence scoring systems increases the validity and reliability of these fidelity measures, consistent with Gearing et al.’s recommendations [22].

For this study, the qualitative data was analysed using Ritchie and Spencer’s framework analysis approach [37] with data deductively mapped to applicable moderating factors described in Hasson’s conceptual framework for implementation fidelity. Co-authors contributed to data analysis and interpretation with NVivo used to support data management and assist with clear audit trail documentation [38]. Qualitative data were reported according to the Consolidated Criteria for Reporting Qualitative Research (COREQ) checklist [39]. Please see Additional file 2 for further details.

Data integration for this convergent parallel mixed methods study was undertaken at the interpretation stage with quantitative, qualitative and then integrated findings presented in this article. The mixed methods findings were reported noting Hadi et al.’s [40] recommendations to improve mixed methods research reporting for pharmacy practice researchers.

Results

Adherence

Range of OSP intervention activities delivered

Within the seven intervention RACFs, the full range of OSP intervention activities were delivered i.e. the online pharmacist diaries indicated that OSPs delivered activities in all nine activity categories. There was one exception to this with one OSP not able to offer vaccination services as they were unable to complete vaccination training due to COVID-19 restrictions affecting training availability. As such, the full range of OSP intervention activities were delivered across each of the seven intervention RACFs as illustrated in Table 2.

Random sample of 10% of medication reviews assessed for quality

Assessment of a random sample of 10% of written medication reviews by two accredited pharmacists using a checklist indicated that the quality of medication reviews across the seven intervention RACFs ranged from high (n = 3), medium (n = 3) through to low (n = 1) as illustrated in Table 2. The rounded mean score for the quality assessment of all written medication reviews was 3 out of 5 indicating an overall medium quality. Assessment of Intraclass Correlation Coefficient (ICC) indicated excellent reliability between the two accredited pharmacists, namely, an ICC of 0.922 (95% CI: 0.697 to 0.974).

Proportion of residents who received at least one medication review

The proportion of residents who received at least one medication review was compared to the PiRACF study a priori activity target of 70% of residents receiving at least one medication review as part of the OSP intervention. OSPs supplied 588 written medication reviews to the study team and 61.1% of residents living across the seven intervention RACFs received at least one medication review as part of the OSP intervention. In this study, using the adherence scoring system, adherence to this fidelity measure ranged from high (n = 4), medium (n = 1) through to low (n = 2), see Table 2.

Overall implementation fidelity rating

Based upon the three quantitative data sets, the overall implementation fidelity adherence score for the seven intervention RACFs ranged from high (n = 1), medium-high (n = 3) through to medium (n = 2) and low-medium (n = 1), as illustrated in Table 2. Overall, it appears that the OSP intervention was generally delivered with medium fidelity across the seven intervention RACFs.

Table 2 Adherence assessment of RACFs
Full size table

Moderating factors

Fourteen interviews were undertaken with RACF managers (n = 7) and OSPs (n = 7 interviews with six OSPs [one OSP worked across two RACFs]). These interviews took between 38 and 163 min, with the median interview duration of 49 min for managers and 148 min for OSPs. Semi-structured interview participant details are shown in Table 3. Most pharmacists had over 10 years professional experience and five were accredited pharmacists. As such, this study was not able to determine if there may be a possible correlation between the level of pharmacist experience and intervention fidelity.

Table 3 Semi-structured interview participants
Full size table

The qualitative interviews provided insights into the following moderating factors outlined in Hasson’s conceptual framework for implementation fidelity: intervention complexity, facilitation strategies, quality of delivery, participant responsiveness and context [30]. Please see Table 4 for the moderating factors, a summary of key findings and exemplary quotes.

Table 4 Moderating factors with a summary of key findings & exemplary quotes
Full size table

Intervention complexity

The qualitative findings suggested that it took time for OSPs and RACF managers to work out how to deliver the OSP intervention, which is not unexpected given the relative novelty of the OSP role and the complexity of the intervention. OSPs considered that the OSP intervention description outlined in the study resource folder was informative and useful in assisting them with delivering this complex intervention. However, some OSPs and RACF managers mentioned that the OSP job description was not as descriptive or instructive as it could have been to support OSP intervention delivery.

Facilitation strategies

A range of facilitation strategies were employed to support delivery of the OSP intervention. These included face-to-face training sessions for OSPs, inclusive of the Residential Aged Care Pharmacist: Foundation Training Program facilitated by the Pharmaceutical Society of Australia (PSA), RACF induction checklist for onboarding OSPs, study team induction meeting with individual RACF managers, as well as with individual OSPs which focussed on OSP orientation, four hour face-to-face quarterly pharmacist support meetings held at the University of Canberra, Microsoft Teams Online Forum to allow OSPs to share information and ask questions in an asynchronous manner and ad hoc individual check ins with OSPs by the study team via face-to-face visits to RACFs and emails. All OSPs attended the half day face-to-face training session and pharmacist support meetings. Overall, OSPs indicated that the facilitation strategies employed, in particular, the pharmacist support meetings, were conducive to the OSP intervention being delivered as intended.

There was however some room for improvement identified in relation to OSP training provided with two OSPs expressing an interest in more palliative care training and self-nominating to attend a Program of Experience in the Palliative Approach (PEPA) training session during the 12-month OSP intervention.

Quality of delivery

Most of the RACF managers interviewed indicated that the OSP intervention was generally delivered to a high standard. With regards to quality of delivery, one key exception related to one OSP who was not able to offer vaccination services to RACF staff as they were unable to complete vaccination training due to COVID-19 restrictions affecting training availability. While all other OSPs offered vaccination services to RACF staff, some OSPs were not able to undertake these services due to vaccination fridge unavailability at two RACFs and due to one RACF using an existing external contractor arranged by their parent organisation. A specific barrier to the quality of delivery identified by four OSPs and two managers related to the part-time nature of the OSP role. One OSP worked across two intervention RACFs participating in the PiRACF study and described how their OSP experience at the first RACF aided them to support OSP intervention delivery with greater ease at the second RACF. This suggested that prior OSP experience was a potential facilitator for delivering the intervention.

Participant responsiveness

Most OSPs and managers engaged well with delivery of the OSP intervention with most participants perceiving that OSPs in RACFs added value and that the OSP intervention should be continued in the future. The qualitative findings also suggested that others, such as RACF staff, were responsive to delivery of the OSP intervention.

However, three RACF managers did identify barriers which contributed to OSP intervention ‘missed opportunities’ within their RACFs. One RACF manager described that the OSP’s limited work day availability and being unwell at the start of the intervention contributed to delays with the OSP establishing relationships with General Practitioners. This meant that it took additional time for the OSP intervention to be optimally delivered.

A second RACF manager perceived limitations on the impact of the OSP intervention due to the OSP at their RACF, who was not accredited, not focussing on medication reviews in the part-time role with limited hours available. The manager also highlighted that the OSP’s reliance upon RACF management to guide OSP intervention delivery meant that the OSP intervention may not have been optimally delivered.

The one exception where an OSP was not able to vaccinate also presented a missed opportunity within that RACF. The manager at another RACF outlined their limited capacity to engage with the OSP to identify opportunities to work together to further optimise delivery of the OSP intervention.

Context

Based upon the semi-structured interviews with RACF managers and OSPs, OSP factors, RACF factors and an external factor may have influenced the fidelity of OSP intervention delivery.

OSP factors

The qualitative findings suggested that RACFs with OSPs who took the time to establish relationships and were pro-active in informing OSP intervention delivery potentially increased the likelihood of the OSP intervention being delivered as intended. Three OSPs also considered that their accredited pharmacist status and experience conducting medication reviews may have helped them to deliver the OSP intervention.

RACF factors

Semi-structured interview participants described how RACF leadership and culture affected OSP intervention delivery. RACFs with management who consistently took the time to work with OSPs to inform OSP intervention delivery and actively supported OSPs within their respective RACFs potentially increased the likelihood of the OSP intervention being delivered as intended. As might be expected, there was a sense that RACFs with a positive culture focussed on collaboration and patient-centred care also increased the likelihood of delivering the OSP intervention as intended.

RACFs spread out over a large area with significant distance between resident dwellings was perceived as a potential barrier to OSP intervention delivery. One OSP who worked across two RACFs perceived that an RACF with this physical environment, as compared to one with a main RACF building, potentially decreased the likelihood of delivering the OSP intervention as intended. It appeared to be more difficult for an OSP to establish and maintain connections with health care team members and residents in the more spread out RACF physical environment without the OSP making a concerted effort to overcome this barrier.

External factor

One external factor was identified as a potential barrier to OSP intervention delivery, namely the COVID-19 pandemic. As the OSP intervention commencement was staggered from April 2020 – January 2021, this meant that the impact of COVID-19 varied across intervention RACFs, though there were commonalities in relation to an overall increased workload on RACF staff and managers trying to mitigate the risk of COVID-19 for residents and RACF staff. One OSP who commenced in 2020 described how the COVID-19 pandemic initially limited their capacity to engage with residents, family members and RACF staff and that it then took some time to re-engage with them.

Integrated findings

The quantitative findings indicated that the OSP intervention was generally delivered as intended with a range of fidelity from low-medium to high and an overall finding of medium fidelity across the seven intervention RACFs. These adherence scores were based upon three quantitative data sets: (1) range of OSP intervention activities delivered; (2) random sample of 10% of medication reviews assessment for quality; (3) proportion of residents who received at least one medication review. Across the seven intervention RACFs, the full range of activities were delivered, there was an overall medium quality assessment of medication reviews and 61.1% of residents received at least one medication review (as compared to the PiRACF study a priori activity target of 70%). The qualitative findings illustrated that the facilitation strategies in place supported OSP intervention delivery, that participants were mostly responsive to the OSP intervention and that the quality of delivery was generally perceived to be of a high standard. Importantly, missed opportunities were identified by three RACF managers which potentially impacted OSP intervention delivery. These included: OSP work day availability which contributed to delays in the OSP intervention being optimally delivered; perceived limited OSP intervention impact due to a non-accredited OSP who was not able to focus on medication reviews during their part-time role whom was also not pro-active in guiding OSP intervention delivery; one OSP not being able to vaccinate; and perceived limited capacity of a RACF manager to work with the OSP to further optimise OSP intervention delivery. Potential barriers (including the part-time OSP role, COVID-19 pandemic, RACFs spread out over a large area with significant distance between resident dwellings) and facilitators (including the study resource folder, pharmacist support meetings, OSPs who took time to establish relationships, experienced accredited pharmacists, RACF managers who actively supported OSPs and worked with them, positive RACF culture) were also identified. Overall, it appears that the medium fidelity of OSP intervention delivery was influenced, either positively or negatively, by a range of moderating factors.

Discussion

This mixed methods study used an established conceptual framework to understand the extent of implementation fidelity and the moderating factors influencing the implementation fidelity of OSP intervention delivery. Prior to the roll out of OSPs within Australian RACFs, it is important to understand whether the OSP intervention can be delivered as intended and what factors moderated intervention delivery. This study found that OSP intervention delivery was implemented with overall medium fidelity across the seven intervention RACFs. Furthermore, several moderating factors contributed to this fidelity, consistent with other health care implementation fidelity studies [25, 29]. The facilitation strategies in place were conducive to delivery of the OSP intervention as intended. Participants were generally responsive and most participants considered that the quality of the intervention was to a high standard. Contextual factors (OSP, RACF and external i.e. COVID-19 pandemic) and the complexity of the intervention itself also impacted OSP intervention delivery. A new and novel contribution of this study was that it identified potential barriers and facilitators to successful OSP intervention delivery in Australian RACFs. More pharmacists with prior OSP experience would likely further support implementation of this intervention in the future.

This study’s finding of an overall medium fidelity rating is relatively comparable to other studies assessing the implementation fidelity of pharmacist interventions in other health care settings [17,18,19]. Sluggett et al.’s mixed method process evaluation of the SImplification of Medications Prescribed to Long-tErm care Residents (SIMPLER) study undertaken in Australian RACFs concluded that their intervention was also generally delivered as intended [16]. The lack of clearly defined adherence measures in health care settings has been previously identified [29] and further consideration of suitable adherence measures to help with standardisation of implementation fidelity assessment for future pharmacist RACF intervention studies is encouraged.

The findings of this study indicated that there were moderating factors which informed OSP intervention delivery. Consistent with van der Laan et al.’s study [18], which assessed implementation fidelity of an intervention in Dutch community pharmacies, this study’s participants suggested that the intervention generally added value with participants mostly responsive to intervention delivery. Facilitation strategies, inclusive of training provided to pharmacists in this study were important for successful intervention delivery, similar to other pharmacist intervention studies which have assessed implementation fidelity [17, 18]. Akin to En-Nasery-de Heer et al.’s mixed method study [17] which explored implementation fidelity of a pharmacist-led intervention involving Dutch community and hospital pharmacists, this study also identified potential barriers for intervention delivery in relation to intervention complexity and time constraints.

According to Hasson, the more clearly defined and described the intervention, the higher the likelihood of fidelity [30]. As such, it is recommended that future efforts to adopt OSPs in Australian RACFs include additional documentation supportive of OSP intervention delivery, particularly in relation to operationalising the OSP intervention which could potentially include initial and ongoing promotion of the OSP role. In addition, tailored support to OSPs and RACF management to facilitate more consistent delivery of the OSP intervention such as through development of additional checklists and guidance documents may be beneficial. Further exploration of the extent of OSP full-time equivalent employment required, particularly within RACFs spread out over a large area with significant distance between resident dwellings, to support effective delivery of the OSP intervention is required.

OSPs highly valued the pharmacist support meetings which enabled them to share their experiences and insights with each other. While the role of pharmacists in Australia has expanded beyond community and hospital pharmacy into Aboriginal Community Controlled Health Organisations (ACCHOs) [41], General Practices [42] and now into RACFs [10], there are limited opportunities available for these pharmacists to meet and connect with other pharmacists working in similar roles. Current avenues which OSPs could access to connect with other non-community and non-hospital pharmacists are limited to the Pharmaceutical Society of Australia’s Interdisciplinary Team-Based Care Community of Speciality Interest [43]. It is recommended that there be further consideration of options to support future OSPs working in Australian RACFs to sustainably connect with each other now and into the future.

In anticipation of the roll out of OSPs within Australian RACFs it would be highly beneficial to consider the overall educational needs of pharmacists commencing in this recently created role. As there is sparse literature on the educational needs of OSPs working in RACFs, in time, when there is a body of experts with expertise on OSPs within RACFs, it is strongly encouraged that consensus be reached on OSP educational needs through undertaking either a Nominal Group Technique or Delphi Technique [44]. Benson et al.’s Delphi study provides an instructive templar on Australian General Practice Pharmacist educational needs [45]. In the interim, it may be useful for OSP educational needs to be guided by the recommended qualifications, skills and training requirements outlined for pharmacists integrated into ACCHOs, General Practices and RACFs within the PSA Pharmacists in 2023: Roles and Remuneration document [46], alongside completion of the PSA Residential Aged Care Pharmacist: Foundation Training Program (or equivalent). Further exploration of the minimum level and extent of pharmacist experience required to effectively support delivery of the OSP intervention is needed.

As identified by Tait et al., pharmacists can contribute to interprofessional collaborative care for older people living in the community and RACFs as they near their end of life [47]. As such, including additional palliative care training to develop OSP skills in end-of-life medication management discussions with residents and family members is strongly encouraged. At a minimum, it is recommended that OSPs should complete the Palliative Care Online Training developed based upon the palliAGED online resources [48] and attend PEPA training session to further understand the use of medications at end-of-life and become more confident in discussing end-of-life care with residents, family members and RACF staff. The potential role of OSPs in supporting residents with end-of-life care needs alongside health care team members and family members, should also be further explored.

Consistent with Choi et al.’s mixed method study [23] which explored implementation fidelity of a person-centred complex intervention in South Korean nursing homes, RACF culture appeared to impact intervention delivery in this study. Given that the OSP intervention was implemented within an existing RACF health care team and culture, we would reaffirm Palmer et al.’s recommendation that organisation readiness be assessed before implementing interventions [24]. One tool which could be employed before implementation of an OSP intervention within Australian RACFs is the Organisational Readiness to Change Assessment (ORCA) [49]. The ORCA tool could potentially help to increase implementation fidelity with its specific consideration of contextual measures (such as staff culture, senior leadership culture) and facilitator measures (such as project communication, planning and team roles to support intervention delivery) [49].

This is the first study that has evaluated the implementation fidelity of an OSP intervention delivered within Australian RACFs. This study demonstrated that the OSP intervention could be delivered with medium fidelity across seven intervention RACFs and reaffirmed the importance of understanding moderating factors which could help to identify barriers or facilitators to successful OSP intervention delivery.

Policy makers, primary health networks, peak pharmacy organisations, researchers, health professionals and RACF management are strongly encouraged to consider the findings of this study and recommendations made prior to the roll out of OSPs in Australian RACFs. These study findings could help inform future efforts to address potential barriers and enhance potential facilitators for successful adoption of OSPs in real world RACFs.

Strengths and limitations

Strengths of this study was the use of mixed methods study design, an existing implementation fidelity framework and development of objective fidelity scoring systems.

Study limitations related to the qualitative findings not being generalisable to other non-metropolitan RACFs. As direct observations by the study team were not conducted and the quantitative activities data was self-reported, potential data accuracy issues may exist [50].

Conclusion

This mixed methods study concluded that the OSP intervention was generally delivered as intended across the seven intervention RACFs with an overall medium fidelity rating. OSP intervention delivery was affected by a range of moderating factors, specifically, intervention complexity, facilitation strategies, quality of delivery, participant responsiveness and context. A number of potential barriers and facilitators to successful OSP intervention delivery were also identified. The findings of this study have important implications for the roll out of OSPs in Australian RACFs and further OSP research studies.