Introduction

A commitment to safe healthcare is a policy goal of governments across the world. However, progress on delivering on this aspiration has been modest, with patients still suffering avoidable harm [1] and rates of harm remaining unchanged over time [2]. The impact that valid and reliable safety data can have on improvement is clear from many domains of healthcare [3]. Although safety data is complex and multi-faceted, it is vitally important to reducing patient harm [4]. This data is required to support the meaningful comparisons between the safety performance of different healthcare organisations and the assessment of the impact of safety interventions [1].

A recent review found that there has been relatively little research carried out on measuring and monitoring safety (MMS) in the healthcare system of the Republic of Ireland [5]. However, there is a recognition that MMS is central to patient safety improvement. This is evident in the Irish Health Service Executive (HSE) patient safety strategy which includes a commitment to ‘using information to improve safety’ (p.8) [6]. Recognising the complexity and multifaceted nature of the MMS in healthcare, Vincent et al. developed the MMS framework [4, 7]. The dimensions of the MMS framework are as follows:

  1. 1.

    Harm: has patient care been safe in the past? (e.g. national audits).

  2. 2.

    Reliability of safety critical processes: are our clinical systems and processes reliable? (e.g. monitoring of vital signs).

  3. 3.

    Sensitivity to operations: is care safe today? (e.g. safety walk-arounds).

  4. 4.

    Anticipation and preparedness: will care be safe in the future? (e.g. safety culture assessment).

  5. 5.

    Integration and learning: are we responding and improving? (e.g. aggregated analysis of incidents) [4, 7].

The framework has been used previously to categorise studies in systematic or scoping reviews of MMS in primary care [8], prehospital care [9], and secondary care in Ireland [5] and Saudi Arabia [10]. The study reported in this paper uses Vincent et al.’s [4, 7] MMS framework to classify the methods of MMS used within secondary care in Ireland. The aims of the study reported in this paper are to (1) examine how patient safety is measured and monitored in Irish hospitals; (2) map the methods of MMS in these hospitals onto the five dimensions of Vincent et al.’s [4, 7] MMS framework; and (3) reflect on the approaches used to MMS in Irish hospitals.

Methods

Research design

A qualitative descriptive approach was adopted for this study to: (1) support a document analysis of the guidance on MMS used in Irish hospitals; and (2) use semi-structured interviews to explore stakeholders’ perceptions about how patient safety is measured and monitored in Irish hospitals. This approach was based upon a study of methods of MMS use in Saudi Arabian hospitals [11].

Phase one: document analysis

Methods of MMS in Irish hospitals that are described in national healthcare governance documents were identified and classified. Document analysis is a systematic method to review or evaluate documents [12]. The ‘ready materials, extract data, analyse data and distil (READ)’ approach [13] was utilised.

Inclusion criteria

The inclusion criteria required that documents were: national-level documents; explicitly discussed or described how patient safety is measured and monitored in Irish hospitals; produced by a national government agency/or an organisation affiliated with a national government agency; and written in English. No publication date or period was specified. Finally, in cases where a document had multiple versions, only the latest version of the document was included.

Exclusion criteria

Documents were excluded if: they did not describe how patient safety is measured and monitored in Irish hospitals; were not produced by an Irish government agency or an organisation affiliated with a national government agency; were focused on tracking performance progress (e.g. annual reports); were focused on the safety of one process only (e.g. medication safety); were focused on a particular method of measurement that is relevant to specific clinical practises (e.g. clinical audit); or were not written in English.

Search process

The search for relevant documents was completed in January 2022 and consisted of four steps intended to support the retrieval of government reports and policy documents:

  1. 1.

    An advanced google search was completed.

  2. 2.

    A search of the following electronic databases was conducted: Medline, CINAHL, OAIster, WHO IRIS, Lenus, and Google Scholar using various combinations of terms ‘measuring safety’, ‘monitoring safety’, and ‘measurement of safety’ (additional File 1 presents the search strategy used in complete detail).

  3. 3.

    Searches were conducted across the Irish Health Service Executive (HSE), Irish Department of Health, and Irish Health Information and Quality Authority (HIQA) websites using both their relevant search boxes and manual search.

  4. 4.

    Potential further related documents were identified through hand searching the reference lists of documents that met the inclusion criteria.

Document selection

The initial screening was completed by YK using the inclusion criteria to assess the potential for inclusion from the titles, abstracts, and/or executive summaries. Documents that appeared relevant were then downloaded for full-text review. A full-text review was completed by YK and POC to ensure the document met all items in the inclusion criteria. Decisions regarding the inclusion or exclusion of documents were agreed by consensus. All decisions were recorded in an Excel file.

Document analysis

YK and POC independently searched through each document and extracted all described methods of MMS. Only minor differences were found between the information extracted by the two reviewers. These differences were concerned with whether a particular method was one measure or could be split into two measures. Once the final list of MMS had been identified, YK, POC, and SL reviewed each measure and reached a decision by consensus as to which dimension of Vincent et al.’s [4, 7] MMS framework it addressed.

Phase two: semi-structured interviews with key stakeholders

The aim of the second phase of the study was to explore what key stakeholders know about how safety is measured and monitored in the Irish healthcare system.

Ethical review

The study was approved by the Clinical Research Ethics Committee, Galway University Hospitals (Ref: C.A.2604). All participants provided signed written informed consent.

Sampling and recruitment of participants

Participants for the interviews were drawn from three different stakeholder groups: (1) policy makers; (2) medical doctors; and (3) nurses. Recruitment of participants was through a combination of purposive and snowball sampling.

Development of interview guide

The semi-structured interview guide that was used is shown in Table 1. The design of the interview guide was derived from the five dimensions of Vincent et al.’s [4, 7] MMS framework. The interview questions were prepared in accordance with best practises for the formulation of interview questions [14, 15]. The interviews were conducted using an interview schedule developed to obtain information about perceptions of MMS in Saudi Arabian hospitals [11].

Table 1 Interview guide used to engage participants in discussion around measuring and monitoring safety in Ireland
Full size table

Procedure

All interviews were carried out from January 2022 to June 2022. After receiving written informed consent, YK and DM conducted the interviews via video conference call. The audios of the calls were recorded.

Interview analysis

The purpose of the interview analysis was to identify the methods of MMS which the participants knew were being used in Irish hospitals. These methods of MMS were categorised using Vincent et al.’s [4, 7] MMS framework. The methods of MMS described by the interviewees were extracted from the transcripts by YK and DM and then reviewed by POC. Decisions on categorisation were made by consensus.

Results

Phase one: document analysis

A total of six documents were found to meet the inclusion criteria. All six documents had been published since 2008. The search process for identifying documents that met the inclusion criteria is shown in Fig. 1. A summary overview of these documents is provided in Table 2.

Fig. 1
figure 1

Flow diagram depicting study selection for document analysis

Full size image
Table 2 A summary of key information of each included document
Full size table

A total of 162 methods of MMS were identified across the six documents (see Table 2 and Additional File 2 for a list of these methods and how they were classified). Of these MMS methods, 30 (18.4%) were concerned with past harm, 40 (24.5%) were concerned with the reliability of safety critical processes, 16 (9.8%) were concerned with sensitivity to operations, 28 (17.2%) were concerned with anticipation and preparedness, and 49 (30%) were concerned with integration and learning. One method of MMS addressed two of the safety dimensions (past harm and integration and learning); therefore, the percentages are calculated out of 163.

Phase two: semi-structured interviews with key stakeholders

The mean duration of the interviews was 25 min (SD = 11 min 58 s). The 24 participants included 18 frontline healthcare workers (nine doctors and nine nurses) and six healthcare policy makers. Of the 24 participants, 14 were women and 10 were men. The participants reported a mean of 13 years of professional experience (range = 3–31 years). Twenty-one (87.5%) of the participants worked in teaching hospitals, and three (12.5%) in national health regulation organisations.

The MMS methods reported by interviewees are shown in Table 3. Illustrative quotes pertaining to the most commonly reported measures in each safety domain are provided in Table 4. The interviewees described a total of 76 methods of MMS. Of these methods of MMS, 14 (18.4%) were concerned with past harm, 25 (33%) were concerned with the reliability of safety critical processes, 8 (10.5%) were concerned with sensitivity to operations, 15 (19.7%) were concerned with anticipation and preparedness, and 14 (18.4%) were concerned with integration and learning.

Table 3 Methods reported by participants to measure and monitor patient safety in Irish hospitals
Full size table
Table 4 Example quotes from the interview transcripts
Full size table

The most frequently reported MMS method for past harm was incident reporting (mentioned by 19; 79.2% of the interviewees). In addition, incident reports were by far the most commonly reported method of MMS across all dimensions. Clinical audit was the most frequently mentioned MMS method for the reliability of safety critical process dimension (mentioned by 18; 75% of the interviewees). Observation and conversations with clinical teams were the most frequently described MMS method for the sensitivity to operation dimension (mentioned by 8; 33.3% of the interviewees), followed by safety walk-arounds (mentioned by 6; 25% of the interviewees) and safety huddles (which were also mentioned by 6; 25% of the interviewees). Analysis of incidents and feedback leading to the implementation of safety lessons (mentioned by 12; 50% of interviewees) was the most often reported MMS method in the dimension of integration and learning.

Discussion

A fundamental challenge to improving patient safety in healthcare is a dearth of high-quality information that allows organisations and individual practitioners to analyse their performance, define priorities, and identify areas of deficiency and risk. In this study, we examined how patient safety is measured and monitored in Irish hospitals, mapped these methods onto Vincent et al.’s [4, 7] MMS framework, and reflected on the meaning of these findings for MMS in Irish hospitals.

Considering both the findings from the document analysis and interviews, it can be seen that a wide variety of methods are used to MMS in Irish hospitals. However, although there were measures from across all five of the MMS framework dimensions, there was some variability in the number of methods within the dimensions. Measures of the reliability of safety critical processes were the most commonly identified methods of MMS in the document analysis and interviews. This may reflect the amount of routine safety data that are collected in Irish hospitals. The dimension with the smallest number of measures was sensitivity to operations. A possible explanation for this finding is that these measures tend to be qualitative (e.g. talking with patients and staff, observing staff) [16]. Such ‘soft intelligence’ is generally more difficult to collect, and analyse, than is the case for quantitative data [17]. However, this qualitative data can provide valuable insights into issues that may not be possible to gain from quantitative methods of MMS. Automated language analysis methods are beginning to be used to analyse qualitative patient safety data [18]. Therefore, there is the potential for the analysis of qualitative data to become much easier, and faster, than in the past.

Of all the measures identified by the interviewees across the five MMS safety dimensions, incident reports were the most common method of MMS—identified by almost four-fifths of the interviewees. However, the international literature suggests that healthcare organisations may overly rely on the analysis of past events and past harms as a source of safety performance information [19,20,21]. Thus, whilst it is positive that frontline healthcare workers are aware of the importance of collecting information on adverse events, it is important that the limitations of this particular measure are recognised. It is well known that reporting systems underestimate the prevalence of patient safety incidents [22] and overestimate the severity of harm [23]. Therefore, it is important to avoid an over-reliance on incident reports and past harm more generally, as the primary source of safety data.

The second most common method of MMS identified by the interviewees was clinical audit—identified by three quarters of the interviewees. Clinical audits are widely used in many healthcare systems to assess clinical performance against pre-set standards and use the data to improve practise [24]. Nevertheless, despite their widespread use, audits’ effectiveness as a practise-improvement strategy is often presumed, rather than supported by robust evidence [24]. A review of the impact of clinical audit on healthcare workers practise and patient outcomes concluded that audits resulted in a little change, only 4% of the studies which used clinical audit resulted in an increase in the desired practise [25]. Research found that audits were more likely to be successful when there was low baseline performance, feedback was offered several times by a colleague or supervisor in both verbal and written formats, and defined objectives and an action plan were included [24]. Moreover, clinical audit places a considerable burden on staff to complete [26]. It is certainly not suggested that the health service abandons the practise of clinical audit. However, there is a need to consider how to reduce the resources, and burden, of MMS [3].

Recommendations

Our study has shown that there are methods of MMS from across all five dimensions of the framework. However, despite collecting large volume of safety data about hospital care, it still remains challenging to determine the safety of the delivery of care [4, 7]. It has been suggested that healthcare stakeholders could get the information they need with a quarter of what is currently being spent on MMS [27]. The WHO has also identified the burden of collecting and analysing data as a barrier to MMS [3]. Therefore, to improve patient safety in the Irish healthcare system, we recommend a number of issues that should be addressed.

  • Reliability of safety data. The reliability of most safety data is unknown, and in some cases the reliability may actually be known to be problematic (e.g. hand hygiene compliance). If measures are poorly designed, this can lead to ‘gaming’, where targets are achieved but the intended changes in practise are not [28]. Therefore, consideration needs to be given to identifying which methods of MMS result in reliable data.

  • Fragmentation of data. There are a huge range of methods of MMS focused at different levels of a healthcare organisation (e.g. units, hospitals), by different organisations (e.g. HIQA, Department of Health). This fragmentation of data creates challenges for healthcare professionals and managers in identifying where improvement efforts should be made, and whether these efforts are effective [1]. It is recommended that there is a consolidation of efforts across these agencies to avoid repetition and overlap of efforts.

  • Quantity of safety data. A total of 162 methods of MMS were identified from the document analysis, and 76 methods of MMS identified from the interviews. This quantity of data can be overwhelming for healthcare workers and managers. There is a need for safety data to be readily interpreted so that safety issues can be identified at unit, hospital, and national levels. Measures that are too burdensome or lack credibility may alienate clinicians and lead to confusion about the impact of interventions [29]. It is suggested that the perspectives of all stakeholders in healthcare should be taken to identify key measures, from across the five MMS domains, that are particularly useful in supporting action and improvement, and do not place a large burden on healthcare staff to use.

  • Lack of ownership of the data. Much of the data is focused on measures generated externally to a clinical team, so the teams may not perceive the data as being related to their performance [30]. Consideration should be given to how to engage front-line clinical staff in MMS so that they feel some ownership and are empowered to act upon the data, and involving them, and other stakeholders, in identifying meaningful methods of MMS.

Limitations

There are a number of limitations of this research that should be acknowledged. The main limitation is that this paper only focused on MMS in the Irish healthcare system. However, the findings are similar to those derived from a study that utilised the same methodology to consider the MMS in the Saudi Arabian healthcare system [11]. Therefore, there would appear to be generalisability of the findings to other healthcare systems. As is the case with other qualitative approaches, our study could be critiqued due to the subjectivity of this type of research. However, these issues were mitigated through the rigorous approach we took to the data collection and analysis. The focus of the interviews was on identifying the methods of MMS, and did not include an analysis of the quality of the data collected using these approaches. Finally, we only considered national-level publications in the document analysis, which may have led to the exclusion of useful hospital-level documents. However, the difficulty in systematically accessing hospital-level documents was a barrier to their inclusion.

Conclusion

There are a wide range of methods of MMS in Irish hospitals. However, having larger numbers of methods of MMS does not necessarily correspond to a robust safety surveillance system. It is suggested that the input of all stakeholders in healthcare are gathered to identify particularly key measures, across the five MMS domains, in order to identify those methods of MMS that are particularly useful in reducing harm, supporting action and improvement, and do not place a large burden on healthcare staff to use or interpret.