Background

Many patients in intensive care units (ICUs) need palliative care [1,2,3] to improve their quality of life as well as that of their families as they all face physical, psychological, social, or spiritual challenges associated with life-threatening illnesses [4]. In ICUs, approximately 20–70% of patients experience physical distress, such as pain, dyspnea, thirst, and fatigue [5], whereas 30–60% experience psychological distress, such as anxiety, depression, and post-traumatic stress disorder [6, 7]. Additionally, patients experience social and spiritual distress, such as loss of social roles, fear of death, loneliness, and loss of self-control [8]. Besides the patients, their families also experience psychological distress, including anxiety and persistent grief disorder [9, 10]. Therefore, quality ICU care should provide palliative care. Basic palliative care for ICU patients can shorten ICU stays, reduce the use of non-beneficial life-sustaining therapies, and reduce psychological distress in patients' families, without shortening patients’ life expectancy [11, 12]. High-quality palliative care can improve the quality of life and symptoms of patients and their families, and may reduce healthcare costs by aligning care with end-of-life goals [11, 13,14,15,16].

Quality indicators (QIs) are statements that define the quality of a service explicitly and quantifiably [17]. Measuring and reporting quality of care using QIs can identify potential problems in care and serve as foundation for further improvement [18, 19]. Previous studies, mainly conducted in the United States, developed and evaluated QIs using electronic medical record (EMR) data [20,21,22]. EMR data have the advantage of reflecting the patient's condition and treatment in an actual medical setting and can be collected without burdening the patients, their families, or clinical staff, as they are accumulated from daily clinical practice [23]. Additionally, discussing and decision-making regarding patients’ values and goals of care is important for quality palliative care [24].

A QI set needs to be based on Donabedian’s theoretical framework, evaluating three aspects including structure, process, and outcome [25]. However, the QI sets developed to date have limitations. Most of them focus on process indicators and do not include all three aforementioned aspects. Second, their feasibility has not been tested during development [26]. Additionally, they are based on a consensus among project teams in the United States [26] and have not been considered for use in other countries, such as those in Asia. Since the roles and functions of ICUs vary according to the healthcare delivery system and culture of each country, it is imperative to develop QI sets for individual countries [27]. Ideally, QI sets should be evidence based; however, evidence of palliative care practices in the ICU is currently limited. Therefore, the present study aimed to develop a QI set for palliative care in ICUs, based on a systematic review and modified Delphi method, which is a formalized process of consensus building by expert groups, and to test its feasibility by reviewing EMR data.

Methods

Study design

The study was performed in two phases, namely the development of QIs using the modified Delphi method and the measurement of quality of palliative care in ICUs by reviewing EMRs.

Development of QIs using the modified Delphi method

The modified Delphi method is a survey technique that involves multiple rounds to reach a consensus and is effective in determining expert consensus even when there is little or no conclusive evidence [28, 29]. The methodology and reporting of the modified Delphi study were based on Recommendations for the Conducting and Reporting of Delphi Studies [30].

Figure 1 outlines the Delphi process conducted between December 2021 and July 2023.

Fig. 1
figure 1

Modified Delphi process. This figure outlines the Delphi process conducted between December 2021 and July 2023

Full size image

In round 1, a systematic review of the QIs of palliative care in ICUs was conducted [26], and focus group interviews were held with eight core study members (palliative care physicians, intensivists, emergency physicians, nurses, and an expert on Delphi methodology).

In round 2, results of the systematic review and focus group interviews were considered, and the National Consensus Project Clinical Practice Guidelines for Quality Palliative Care, 4th Edition [31] was referred to in order to identify the best practices for palliative care of ICU patients and develop candidate QIs based on Donabedian's structure-process-outcome framework [25].

Subsequent rounds (rounds 3–6) comprised questionnaire surveys among expert panels, teleconferences, and e-mail discussions among the core members. One core member (Y.T.) sent an e-mail inviting potential panel members to participate in the study. The panel consisted of 16 members (five physicians, eight nurses, one physical therapist, one pharmacist, and one medical social worker) specialized in palliative or intensive care. Additional file 1 presents the details of the survey's rating by the panel of experts.

Pilot test of QIs for palliative care in ICUs via EMR review

The QIs for palliative care in ICUs, developed in this study, were used to measure the QIs by reviewing EMRs.

The two sites included in this study were a general ICU and an emergency ICU within the same university hospital. The general ICU is a semi-closed ICU system that is primarily intended for patients with acute illnesses, postoperative patients, and patients with chronic illnesses such as chronic respiratory disease, heart failure, and renal failure. The emergency ICU is a closed ICU system that is primarily intended for emergency patients, including those with trauma, stroke, and other conditions requiring immediate medical intervention. The general ICU had 18 beds, and the emergency ICU had 16 beds. Both ICUs have intensivists on duty 24/7. The hospital is the largest in the region, with approximately one million patient visits per year. The hospital's ICUs have been designated as formal training facilities for intensivists.

Eligible patients were defined as follows: (1) patients admitted to the ICU on an urgent basis, and who stayed in the ICU for 48 or more consecutive hours or (2) patients admitted after scheduled surgery, who required mechanical ventilation for more than 48 consecutive hours. These patient criteria were established through a Delphi process to develop the QIs. We reviewed the EMRs of patients admitted to the ICU between January 1, 2019, and June 30, 2019. Data were collected from core-member nurses who were not working in the ICU. To verify the inter-rater reliability, another nurse independently reviewed the QIs and checked agreement with the main reviewer. We randomly selected 10% of the patients from our sample for testing. Data collectors also monitored the time required to measure the QIs per patient.

Each process and outcome QI was assessed at the individual level over the duration of ICU care. Aggregate quality scores were calculated by dividing the number of times palliative care was provided (numerator) by the number of eligible events (denominator). Structural QIs were assessed by reviewing the site policies and interviewing the site staff. Data collected included patient demographic data, survival data, disease severity, and length of ICU stay. A QI measurement manual was developed and used (Additional file 2).

Ethical approval for this study was granted by the Institutional Review Board of Tohoku University (No. 2022-1-1023), and the study conformed to the principles outlined in the Declaration of Helsinki.

Statistical analysis

Summary statistics were calculated, and frequency distributions of the data were stratified according to each ICU. Performance frequency was calculated for each QI value. Continuous variables are presented as mean and standard deviation (SD), if normally distributed, and as median and interquartile range, if not normally distributed. To assess inter-rater reliability, we calculated the Cohen's kappa coefficient and the mean of differences. The data were analyzed using JMP version 17 (SAS Institute, Cary, NC, USA) and R version 4.2.0 (R Foundation for Statistical Computing, Vienna, Austria).

Results

Development of QIs using the modified Delphi method

In round 1, a systematic review was conducted, and 109 QIs were extracted from 5 literature sources [20, 22, 32,33,34]. The results were combined with the opinions from the focus group interviews to develop a draft of 35 candidate QIs.

In round 2, the core members discussed the appropriateness of each QI as an indicator to assess high- or low-quality palliative care in the ICU and narrowed the list to 28 candidate QIs.

The expert panel consisted of 16 participants with 20.0 ± 7.2 years of experience. Response rate of the surveys (rounds 3 and 5) was 100%; none of the responses were missing.

The panel members rated each QI item in rounds 3 and 5 (Table 1). In round 3, there was no item with a median score below seven for appropriateness, but five items (items 3, 5, 7,8, and 13) were rated low for feasibility. The core members mainly discussed feasibility issues (round 4), revised the QI titles, and modified the denominator settings. In round 5, feasibility ratings improved for most items. However, the following two items had median scores below seven: item 5, assessment of the patient's psychological distress, and item 7, assessment of the patient's spiritual and cultural practices.

Table 1 Delphi ratings for the advocated quality indicators
Full size table

Discussions among core members (round 6) resulted in opinions regarding two of the above indicators, such as "the low feasibility of measurement ratings may simply reflect the fact that they are not currently being documented." Another commented, "spiritual and cultural practices are important to understand what kind of person the patient is, and are necessary as a consideration for diversity in the future." Therefore, we agreed that the two indicators would be necessary to improve palliative care in our country in future and decided to adopt the two items: item 5, assessment of the patient's psychological distress, and item 7, assessment of the patient's spiritual and cultural practices. Consequently, we modified the first 28 candidate QIs, and the new 28-item QI set consisting of eight domains became the final version (Table 2).

Table 2 Set of 28 quality indicators for palliative care in ICU
Full size table

Pilot test of the QIs for palliative care in ICUs via EMR review

Pilot testing of the developed QI set confirmed that process and outcome indicators were measurable from EMR data, whereas structural indicators were measurable from a survey of facility policies. Inter-rater reliability of the assessments was evaluated using Cohen's kappa coefficient. Overall, for QIs 5–15, the obtained Cohen's kappa value was 0.92 (95% confidence interval [CI] 0.87–0.97), indicating substantial inter-rater agreement. Additional file 3 presents the inter-rater reliability for each QI. Measurements per patient took a mean time of 32.2 ± 16.8 min.

Patient characteristics

In 2019, a total of 1697 patients were admitted to the ICU. Of these, 850 patients were admitted from January 1, 2019, to June 30, 2019, and 262 (30.1%) were eligible for measurement of the QIs. Of the eligible patients, the patients scheduled for surgeries were all admitted to the General ICU, which accounted for 23.1% of the total number. The mean age of the patients was 63.1 ± 17.6 years, and 42.4% were admitted to the ICU for surgery. The mean duration of ICU stay was 7.1 days (Table 3).

Table 3 Characteristics of ICU patients
Full size table

Structures, processes, and outcomes of palliative care in ICUs

Table 4 presents the frequencies of process performance and outcome indicators for the entire study sample. Of the 15 process and outcome indicators, 7 with performance frequencies less than 50% were as follows: (1) regular pain assessment, (3) reassessment of pain after treatment and/or management, (5) assessment of patient’s psychological distress, (7) assessment of patient’s spiritual and cultural practices, (8) identification of the patient’s advance directive and advance care planning (ACP) for treatment, (9) conducting an interdisciplinary family conference on palliative care, and (11) assessment of family members’ psychological distress. Additionally, there were some differences in the percentage of practices across the sites. The largest differences between the sites were regarding (1) regular pain assessment and (14) pain-free status in the last 24 h of life. “Patient pain-free in last 24 h of life” could not be evaluated due to the low implementation rate of “regular pain assessment” in emergency ICU.

Table 4 Frequency of performance of the measures of processes and outcomes of palliative care in ICU
Full size table

Table 5 presents the results of evaluation of the structural QIs of the two ICUs. The results were similar since the ICUs were located within the same hospital.

Table 5 Performance frequency of measures of palliative care structures in the ICU
Full size table

Discussion

In this study, based on a systematic review and expert consensus, we developed a QI set for palliative care in ICUs that consisted of 28 items across eight domains. Pilot testing demonstrated the set of QIs to be feasible and measurable. Additionally, the pilot test results suggested several potential improvements in palliative care in ICU settings.

This study developed a QI set that followed Donabedian’s structure-process-outcome framework. Additionally, the QI set was based on a systematic review and covered eight domains of the Clinical Practice Guidelines for High-Quality Palliative Care, which provide a foundation for improving the quality and delivery of palliative care in the United States [31]. In the Delphi rounds of this study, two items, namely “assessment of the patient's psychological distress” and “assessment of the patient's spiritual and cultural practices,” were rated as having low feasibility. Low-feasibility indicators are often not documented in EMRs, and hence, the potential need for quality improvement is missed [35, 36]. We included these two items in the QI set, assuming that low feasibility due to non-documentation is indicative of poor quality of care.

This study confirmed the feasibility and inter-rater reliability of QI measurements using medical record data from two ICUs. An average time of 32 min was spent on each patient to measure the process outcome QIs. Since this was a pilot study, the time taken for these measurements may reduce as people become accustomed to the task of measurement. A study testing the quality-of-care indicators for patients with cancer reported that the review of 92 indicators from EMRs takes approximately 2.4 h per patient [37, 38]. Thus, the necessary information may be collected within a reasonable time, but the simplicity and feasibility of the evaluation method should be further enhanced to promote quality palliative care worldwide. In this set of QIs, the evaluator spends more than half of the time on the analysis of the patient's assessment of pain and delirium. In this regard, to reduce the evaluation time, data covering only 48 h after admission to the ICU could be assessed. In addition, natural language processing and machine learning techniques have been demonstrated to handle textual data from EMRs with little burden on healthcare providers or patients in palliative care settings [39,40,41]. This state-of-the-art technology can enable rapid review and feedback of documentation. Therefore, the simplification of QI measurement methods and the use of technology in evaluation are important to increase the feasibility of continuous evaluation.

Pilot testing of the medical record review survey identified the following four aspects that required improvement: assessment of regular pain, identification of the patient's advance directive and ACP for treatment, conducting an interdisciplinary family conference on palliative care, and assessment of the psychological distress experienced by family members.

In this study, the frequency of pain assessment was 46%. Previous studies conducted in the United States reported pain assessment frequencies of 76–87% [20, 21], hence, suggesting the need for improved pain assessment in Japan. The outcome indicator “no pain in the 24 h before death” was 69% in this study, which is not greatly different from 72% in the United States [34]. However, because the frequency of regular pain assessment was low, the presence or absence of pain may not have been accurately assessed, thereby possibly affecting the outcomes. The guidelines emphasize that the presence of pain and need for pain management should not be dismissed in critically ill patients, who are often unable to communicate clearly [42, 43]. Therefore, continuous assessment and recording of pain severity would ensure that pain presence is not neglected in critically ill patients; this would improve the quality of ICU care, including the efficient use of analgesics and sedative medications.

In this survey, approximately 10% of respondents confirmed their advance directive and ACP for treatment, and approximately 30% discussed the patient's quality of life and values at a multidisciplinary conference that included family members. This was similar to the results of a survey of three ICUs in the United States, which reported advanced directive status (31%) and conduction of family conferences (19%) [21]. In a national institutional survey of ICUs in Japan regarding the frequency of multidisciplinary conferences on palliative care, 75% of ICUs reported that conferences were never held, and only 8% responded that they were held four or more times a month [44]. Interventions in family conferences and communication are necessary to provide patient-centered care and increase family satisfaction and trust in healthcare providers [45]. In recent years, the decision-making process for treatment in ICUs has recommended the implementation of discussions with patients and their families about goals of care within 5–7 days of ICU admission and holding of weekly multidisciplinary conferences [46]. Confirming ACPs regarding treatment and conducting family conferences are important for achieving treatment goals that are in line with the patient's wishes, that is, to provide patient-centered care.

A unique element of this QI set is the inclusion of a QI that assessed families’ psychological distress. In ICUs, where patients often lack decision-making capacity or are unable to express their treatment preferences, the patient's family is an important component of palliative care. Patients’ families, who are often surrogate decision-makers in the unique environment of the ICU, also experience psychological distress, since they are in crisis [9, 10]. Therefore, family members of ICU patients should be recognized as beneficiaries of palliative care, and assessment of family members' psychological distress is considered important.

The structure indicator in this QI set consists of three domains, namely "Setup and availability of resources and care protocols," "Support system for patient's family," and "Support system for ICU staff." Support for ICU staff is a distinctive domain. Prevention of burnout through a support system of ICU staff is an important indicator that leads to the continuous provision of high-quality care to ICU patients and their families [47]. This survey could not evaluate the structure of palliative care in ICUs in Japan, since data were obtained from only two ICUs. In future, surveys should be conducted at more facilities to clarify the current status at the national level.

Implications for future

This study developed a QI set comprising QIs that can be measured using data extracted from medical records, allowing easy measurability and sustained and continued use of the QIs, with limited measurement of patient outcomes. Future studies are warranted that evaluate patient-family experience outcomes from questionnaires regarding patient health [48] or assess family satisfaction using the questionnaires regarding intensive care units [49] in conjunction with measurement of this QI set. To improve the quality of ICUs, establishment of a system that, in parallel with educational interventions for staff, feeds the results of quality measurement surveys back to clinical practice in a cycle of improvement, would be required [50]. Kruser et al. conducted a large study that enrolled 68 ICUs in the United States and evaluated the three aspects of structure, process, and outcome to identify unit-level variation. To improve quality of palliative care in ICUs in future, they proposed analyzing the potential characteristics of units that provide high-quality care [34]. Even in Japan, where palliative care in ICUs is still developing, a multicenter survey needs to be conducted and benchmarked to assess the quality of care at the national level. Continuous research utilizing this QI set needs to be conducted to monitor the cycle of change and improvement in clinical practice, owing to the implementation of awareness and educational interventions.

Study limitations

This study had several limitations. First, since the measures in this QI set were based on medical record data, we may have underestimated the actual care provided in the absence of documentation. On the other hand, there are concerns about the risk of the possibility that the measures are not actually being provided even though they are being documented in the EMR and the risk of copy and paste duplication of documents [23]. However, we believe that documentation of pain assessment and discussion of goals of care are important in good-quality palliative care and may be quality indicators in and of itself. Second, since the study was conducted in two units within a single hospital, generalization is limited. Different hospital sizes and regions may have different demographic characteristics and disease coverage. In addition, the way in which medical records are written can vary depending on the hospital's situation and organizational culture. To deal with the issue of representativeness of the pilot test, the study was set up in a large-sized university hospital that serves as a teaching facility for intensive care physicians, and thus requires a certain level of medical care. Furthermore, by validating the study in two different types of units, we endeavored to accommodate variations. As a logical next step in the research, a multicenter study is being considered for further validation of our findings. Finally, although we developed a comprehensive QI set related to patient/family-centered and ICU staff-support domains, evidence associating most structures and processes of palliative care in the ICU with outcomes remains limited [51]. This would need to be modified in future to adjust to the changes in practice and accumulate new evidence.

Conclusions

A 28-item QI set was developed using the modified Delphi method and measured using EMR data, thereby providing a tool for assessment of the quality of palliative care in ICUs. Pilot testing using medical record data from two ICUs confirmed its feasibility and measurability. Our pilot study suggested the aspects of palliative care in ICUs in Japan that need to be improved, further highlighting the importance of conducting a nation-wide multicenter survey. To conduct large-scale surveys in future, we need to test the QI set further, with focus on decreasing the burden of measurement.