Background

The Health Service Executive (HSE) in The Republic of Ireland, along with many other international health care providers recognise the need for high quality, standardised data collection that quantifies nursing and midwifery care processes [, 1, 2]. While such data can contribute to improved understanding of mental health nursing care processes, the choice of which mental health nursing metrics to measure presents a significant challenge in Ireland and internationally [3].

Contemporary mental health care policy emphasises the need for measurement of the contribution of mental health nurses to patient safety and clinical quality improvement [4], 5, 6]. Currently in Ireland and internationally there very limited research in this area and as such no quantifiable means of establishing the contribution of mental health nursing processes to the outcomes of people in receipt of mental health nursing care across the lifespan and healthcare settings. Notable historical efforts to identify and quantify what mental health nurses do include the development of clinical indicators for mental health nursing in Australia and New Zealand in the 2000s. While not developing metrics and their associated indicators, Skews et al. [7] and O’Brien et al. [8] produced clinical indicators and endeavoured to audit these as a means of comparing and benchmarking practice across mental health settings. It is noteworthy that these studies acknowledged the challenge of capturing the totality of mental health nursing practice and of embedding such measurement approaches. In a discussion on mental health nursing culture Slemon et al. [9] note that mental health nursing practice remains focused on mitigating risk and promoting safety, while lacking effective structures. Indeed, even in these priority areas, where mental health nurses utilise rigorous monitoring and recording as means of risk management the evidence is that current approaches lack efficacy [10].

In response to the identified need the Office of Nursing and Midwifery Services Directorate in Ireland commissioned a national research study to establish a consensus on how nursing and midwifery care processes that should be measured (HSE 2018a). This national study aimed to develop care process metrics and indicators that where possible would align with evidenced-based clinical practice guidelines and standards. This study produced a suite of seven QCM reports that outline these metrics and associated indicators in the healthcare areas of Mental Health, Midwifery, Children’s Community/Public Health, Acute, Older and Intellectual Disability (HSE 2018b). This paper reports the development and prioritisation of a national suite of QCM, and their associated indicators, for mental health nursing in Ireland.

Methods

This study comprised four discreet phases outlined below. With all phases overseen by a work stream working group made up of mental health nurses from a range of grades, specialities and geographical areas, service user representation and an academic lead (n = 30). The work-stream working group members were invited to participate in the quality care metrics project by letter informing them of the study and inviting them to nominate registered mental health nurses of all grades from their area to participate in the study. The group was provided with regular updates by email and teleconference and met face to face four times over the course of the project; before the Delphi study commenced, during the Delphi study and after the Delphi study had completed, to agree the final suite of metrics.

Phase 1

A systematic literature review to identify mental health nursing metrics that have been used in this area and the indicators for same. The following inclusion criteria were applied; i) Participants: registered nurses or midwives working in any of the seven identified work-streams, or persons in receipt of nursing or midwifery care from eligible work streams; ii) Exposure: relating to nursing or midwifery quality care processes (metrics or indicators). For the purposes of this study a quality care process metric is defined as a quantifiable measure that captures quality in terms of how (or to what extent) nursing or midwifery care is performed in relation to an agreed standard. A quality care process indicator is defined as a quantifiable measure that captures what nurses or midwives are doing to provide care in relation to a specific tool or method (Foulkes et al. 2011 [11]); iii) Outcomes: measurable quality process or processes in use or proposed for use; iv) Type of study: any.

Phases 2 and 3

A two-round online Delphi survey of mental health nurses to develop consensus on metrics to be prioritised and a two-round online Delphi survey of mental health nurses to develop consensus on indicators for the prioritised metrics. At the end of the first two rounds, the metrics were identified and at the end of Round 3 and 4, the indicators for those metrics were identified. A purposeful convenience sample of mental health nurses working in the mental health services in Ireland were invited to partake in the Delphi survey. Inclusion criteria for the study included any registered mental health nurse working in the HSE mental health services in Ireland, with the ability to give informed consent. Exclusion criteria for the study included any registered mental health nurse not working in the HSE mental health services in Ireland. There were 1102 responses across four Delphi survey rounds.

All mental health nurses who met the inclusion criteria for the study were invited to participate in the study following a comprehensive national advertising campaign via posters, information sheets, word of mouth and presentations at national nursing conferences and events. Participants indicated their interest in the study via email to the research team.

The Delphi data collection and analysis provided group consensus on what metrics (Rounds 1 & 2) and associated indicators (Rounds 3 & 4) should be used. Open fields were provided in Rounds 1 and 3 to allow participants to comment on and suggest additional potential metrics and indicators. Responses to each round were collated then redistributed to participants for further comment in successive rounds. Each round had a response closing date of 21 days after the date of invitation. E-mail reminders were sent to anyone who did not respond by day 7 from the date of invitation. Numbers of participants for each round of the Delphi are presented in Table 1.

Table 1 Nursing and midwifery judgement framework tool
Full size table

Phase 4

Face-to-face consensus meeting with key stakeholders to review the findings and build consensus on the final suite of metrics and indicators. Phase 4: Face-to-face meeting with key stakeholders to build consensus on final metrics and indicators.

This phase comprised of a face-to-face meeting with attending members of the work stream working group (n = 20). The work-stream group members were invited to review and build consensus on the quality care process metrics and indicators developed from the Delphi surveys. Participants were provided with a Nursing and Midwifery Judgement Framework Tool adapted from Flenady et al. [12] to guide their determination on metric/indicator inclusion in the final suite of mental health QCMs (see Table 2).

Table 2 Overall Delphi responses by grade and role
Full size table

Ethics

Ethical approval was granted by the Research Ethics Committee National University of Ireland Galway. All potential participants received a study information sheet, which outlined the purpose of the study, the risks and benefits of participation, confirmed participation was entirely voluntary and the likely time commitment. They were also informed they could ask questions at any point and given contact details to that end. All participants had to explicitly indicate their consent to participate by clicking on the ‘I agree’ button at the end of the online participant information sheet before accessing the survey. For phase 4, consensus meeting participants were given a participant information leaflet. Written consent to participate was then obtained at the meeting.

Results

Systematic review

The literature search was undertaken as a national collaboration across all 7 work streams (Devane et all 2019 [13], Doody et al. 2019 [14], Murphy et al. 2019 [15]). The aim of the review was to identify quality care process metrics and their associated indicators, and to identify the current evidence base.

Eight databases were systematically searched including: Pubmed, Embase, PyscINFO, ASSIA, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Cochrane Database of Systematic Reviews (CDSR), Cochrane Central Register of Controlled Trials (CENTRAL), and Database of Abstract of Reviews of Effects (DARE).

To maintain contemporary relevance the search was undertaken between 1st January 2007 and 1st January 2017, in English language where full text were available. For this purpose a systematic review procedure was adapted using the search terms nurs*:ab,ti OR midwi*:ab,ti AND (‘minimum data set’:ab,ti OR indicator*:ab,ti OR metric*:ab,ti OR ‘quality measure*’:ab,ti) AND [english]/lim AND [2007–2017]/py. The search was not limited to study design but widened to comprise all types of sources including grey literature.

Covidence software (Cochrane 2016) was utilised to manage the retrieved studies. After duplicates were removed, each title was reviewed independently by at least two members of the national academic teams. Disputes were settled by discussion and negotiation. At full text screening, any included studies were tagged to the specific work-streams. Full-text studies relevant to each work-stream were then reviewed by two reviewers (NB and AH for mental health nursing) from the appropriate work-stream. The complete process flow diagram for the systematic literature review is presented in Fig. 1. Search and selection flow diagram.

Fig. 1
figure 1

Search and selection flow diagram

Full size image

Systematic review findings

The search conducted across eight databases resulted in 15,304 citations. Following removal of duplicates, 7,524 unique references were identified. Following title and abstract screening, 218 sources were retained, including those identified from hand searching and the grey literature, for full-text screening. Following full text screening, 112 sources were included upon the basis that they met the study’s inclusion criteria. These sources were reviewed for reference to mental health nursing care processes and quality processes (see Fig. 1). Eight studies were identified as relevant to mental health nursing. Two researchers reviewed these independently for quality and content. Disagreements were resolved between the reviewers and a third reviewer consulted if required. Two from the database search [16, 17] both of which referred to measurable approaches to mental health screening, undertaken by mental health nurses. From the grey literature, the Judgement Support Framework [18] a guidance document designed so support quality in mental health care processes implementation and monitoring.

Following the systematic review process, the Mental Health Work Stream Working Group met to discuss the potential metrics extracted from the systematic literature review. At this stage it was agreed that, the pre-existing suite of mental health nursing care process metrics [19] be included in discussion for possible inclusion. Following this discussion, 16 potential mental health nursing metrics were agreed for inclusion in Round 1 of the Delphi survey: 1. Medication Storage and Custody, 2. Management of Controlled Drugs (MDA Drugs), 3. Medication Administration, 4. Ensuring Correct Prescription of Medication, 5. Assessment and Personal Details, 6.Nursing Care Plan, 7.Nursing and Midwifery Board of Ireland (NMBI) Record Keeping, 8. Provision of Required Information for Service Users & Carers, 9. Discharge Planning, 10. Service User Experience, 11. Screening and Evaluation of Mental Health Needs, 12. Use of psychotherapy/nonpharmacological therapeutics i.e. talk therapies, 13. Use of psychiatric medications side effect profile, 14. Care of the dying, 15. Communication of Healthcare Team, 16. Therapeutic Communication with service users.

Phases 2 and 3 The Delphi consensus process

Delphi-round 1

All those who expressed interest in participating were sent email invitations from SurveyMonkey® to participate. A web link was also created as an additional data collector. Round 1 of the Mental Health Metrics Delphi was launched on the 6th of June, 2017 and remained open for 21 Days. The Delphi survey consisted of consent to participate via an: I agree or do not agree button as well as questions regarding demographics. Participants were asked to rate each of the 16 included metrics from 1–9 (1-Not Important, 9-Very Important). SurveyMonkey® participants were updated weekly according to both Community Health Organisation (CHO) area and nursing grade and circulated to project officers and directors of mental health nursing to track participation rate. Following removal of duplicates and inclusion of only those who provided an email address, there were a total of 290 participants in Round 1.

Delphi round 2

All who expressed interest and whose responses provided data from Round 1 were sent email invitations via SurveyMonkey® to participate in Round 2 of the Mental Health Metrics Round 2 Delphi. Participants were also sent confidential emails prior to the start of Round 2 with PDF copies of their individual Round 1 survey responses to allow them to re-rate the metrics based on both their responses and the group’s responses. Round 2 of the Delphi was launched on the 11th of July 2017 and remained open for 21 Days. Following removal of duplicates and inclusion of only those who provided an email address there were a total of 233 participants. Round 2 participants were asked to rate 20 proposed metrics that had either been maintained from Round 1, added from open fields in Round 1 of re-worded following review by the work stream working group.

Delphi round 3

This round of the Delphi survey was open to new participants that had not previously participated in Round 1 or Round 2. These participants were asked to rate 80 potential indicators that could be used to measure the seven metrics retained from Rounds 1 and 2. Email invitations via SurveyMonkey® were sent to those who had previously completed Round 1 or Round 2 in addition to any new expressions of interest. Round 3 of the Delphi was launched on the 22nd of August and remained open for 21 days. Following removal of duplicates and inclusion of only those who provided an email address there were a total of 233 participants.

Delphi round 4

Email invitations for Round 4 of the Delphi survey were only sent to those who provided data in the Round 3 survey. Participants were also sent confidential emails prior to the start of Round 4 with PDF copies of their individual Round 3 survey responses to allow them to re-rate the indicators based on both their responses and the group’s responses. Round 4 of the Delphi was launched on the 3rd of October 2017 and remained open for 21 Days. Following the removal of duplicates and inclusion of only those who provided an email address there were a total of 143 participants.

Delphi rounds data analysis

As noted participants across the Delphi rounds were asked to indicate the importance of the metrics and indicators using a 9-point Likert scale. At the completion of Delphi round 2 (metrics) and Delphi round 4 (indicators) consensus on inclusion was determined where 70% or more participants rated the metric or indicator as 7 to 9 and less than 15% of participants rated the metric as 1 to 3. These criteria were used consistently across the four rounds to best represent the participants’ judgements over the rounds [20]. The 70% importance rate is also consistent with approaches to developing core outcome sets in routine healthcare http://www.comet-initiative.org/.

Phase 4

Consensus meeting participants voted on each metric and indicator as a yes or no to be included using the Poll Everywhere App. Metrics and indicators were required to receive a vote of 70% or higher to be included for the final suite in mental health quality care metrics. Follow-up discussions and multiple rounds of voting were used where necessary. Following this process, the final agreed suite of nine mental health nursing process metrics and their indicators were finalised (see Table 3):

Table 3 Final agreed metrics (n = 9) and indicators (n = 71) with Delphi round 4 consensus percentages
Full size table

Discussion

The 9 metrics and 71 indicators presented in Table 3, represent the main findings of this study. The metrics and indicators included in the final suite show considerable synergy with literature indicating areas where mental health nurses and service users have identified the need to develop the knowledge, skills and practice of mental health nurses [21]. By identifying prioritised areas where mental health nursing processes can be measured this research contributes to efforts to quantify and improve practices that mental health nurses, service users and carers representatives value. This understanding of these stakeholders’ views on key areas of nursing process indicate potential areas for the development of mental health nursing practice and education.

It should be noted that metrics as applied to health care practices arise from the world of business where metrics were devised as means of setting targets and measuring attainment of these [22]. Pencheon [23] suggests that within healthcare settings, metrics as measured by their indicators illustrate areas of practice performance and the degree to which expectations are being met. Hence, these can be used as a quality control measure in practice. Typically, metrics and their associated indicators are used to show benchmarking or attainment against agreed standards, with a view to ensuring and improving the quality of care [, 24,25,26]. It should be noted that while there are clear potential benefits to the implementation of metrics, there is ongoing concern that care processes, in this case the day to day business of mental health nursing cannot be adequately measured [3]. The concern being that the interpersonal day to day activities delivered by mental health nurses are hard to measure and that mental health nurses do not work alone making their activities as distinct from multi-disciplinary team activities hard to isolate and quantity [27].

In this study, despite the lack of applicable evidence based metrics and indicators identified from the systematic review, the grey literature identified valuable starting points for the deliberations of the work-stream working group, in identifying areas of importance to mental health nursing in the Irish context. The literature sources that were identified ranged from full procedure guidelines with some underpinning evidence through to checklists of areas pertaining to practice to be measured.

Central to the conduct of this research and complimented by the rigorous research process outlined is the level of engagement with service user and family representatives and a broad cross-section of mental health nurses across grades, practice settings and geographical areas in Ireland. This ensured that mental health nursing QCMs and associated indicators arose from genuine consensus. Crucially there was service user and family representation from the outset, across the work stream working group deliberations through to the final Phase 4 consensus meeting, ensuring maximum possible involvement in the ongoing research. This approach reflects the policy emphasis in Ireland that mental health service provision should arise from genuine partnership between service users and service providers [28,29,30].

Despite the lack of applicable evidence identified from the systematic review every effort was made to ensure the quality of the mental health nursing metrics and indicators included in the final suite. The evaluation tool used identified four key attributes of metrics and indicators these being process focused, important, operational and feasible. The rigorous research design employing 4 Phases; systematic literature review, online Delphi survey of mental health nurses and consensus meeting with key stakeholders, means that the finally agreed metrics and indicators can be considered to be process focused with clear applicability mental health service users and mental health nurses in practice. Importantly, as noted in the systematic review section, not all of the agreed metrics and indicators have supporting clinical standards and an evidence base although they do have a strong practice evidence base. This understanding specifically impacts on feasibility, the fourth component of the judgement framework tool. This suggests that there is a need for robust piloting and evaluation of these metrics and indicators before rolling out service wide to identify potential unintended impacts and barriers [31]. Consequently, this initial development of metrics and indicators should be followed by a rigorous baseline review of QCM uptake and implementation amongst mental health nurses as part of an ongoing evaluation.

Conclusion

Achieving consensus on what mental health nurses do and how best to measures those processes in an important element of ongoing efforts to improve the safety and quality of mental health nursing practice. Having achieved its aim in producing a suite of mental health nursing metrics and indicators, there is now the opportunity to incorporate routine measurement into ongoing efforts to improve the quality of mental health nursing care. Any implementation of these metrics and indicators could provide valuable evidence of mental health nurses contribution to safe, quality, care processes.