FormalPara Take-home message

Whilst recognition of the need for ongoing investment in health service infrastructure remains, stakeholders seeking to improve quality of critical care in LMICs are increasingly focused on improving processes of care, notably reducing avoidable harms (specifically healthcare associated infection and antimicrobial resistance) and patient centred outcomes. Continuous surveillance using electronic records and registries were perceived as essential infranstructure to faciliate implementation.

Introduction

Intensive care units (ICUs) provide essential and lifesaving interventions to critically ill patients [1]. Delivery of critical care is resource intensive, complex and with considerable burden for both patients, families and those responsible for provision of healthcare services [2]. Whilst increased access to critical care has contributed to a reduction in global mortality from disease and injury, the quality of critical care remains variable internationally. Poor quality care is associated with increased hospital stay, excess morbidity, and avoidable healthcare associated costs [3].

In high-income countries (HICs) and in those where ICU facilities are widely established, there has been an increasing focus on promoting safe and effective delivery of care, monitored through a series of quality metrics (indicators) measuring availability of structures, and the quality of processes and outcomes of care [4,5,6,7]. In general, indicators are derived almost exclusively from the values, practices, and organisational structures seen in a relatively narrow ICU construct synonymous with HIC settings. Consequently, few of these indicators have been evaluated for feasibility, stakeholder relevance or ability to drive actionable improvement in LMIC settings [6, 8, 9].

Efforts to develop globally applicable indicators suitable for LMICs have focused on health system measures, such as those used to benchmark national services and drive Universal Health Coverage [3]. Such metrics often struggle to reflect the priorities of patient and frontline healthcare providers and have had limited success in translating into actionable measures, resulting in under adoption and underutilisation [6, 8, 10,11,12,13]. To address this gap in stakeholder selected indicators that reflect both priorities and practices of critical care services in LMICs, we undertook a RAND Delphi study [14] to develop a set of actionable quality indicators for use in ICUs in LMIC settings. Furthermore, we sought to identify barriers and facilitators to implementation of the selected indicators as perceived by the clinical teams delivering frontline care.

Methods

Study design

This Delphi study is reported in accordance with the Conducting and REporting DElphi Studies (CREDES) guidelines [15]. An advisory committee (AC) was convened to provide oversight, and an Expert Panel (EP) (identified by the AC) was responsible for selecting indicators by voting in the Delphi, highlighting when indicator definitions were problematic, and identifying potential barriers and facilitators to implementation of the indicators. In addition, a subset of the AC and EP were convened into a Definitions Working Group (DWG). The DWG was responsible for the appraisal of the voters’ feedback on the definitions of the indicators, adaptation and refinement of those definitions following each round of voting.

The Delphi was conducted electronically using the online survey tool, Survey Monkey Inc (San Mateo, California, USA; www.surveymonkey.com). Prior to conducting the Delphi, it was piloted for readability, interpretability, and user experience by a physician, a nurse and a non-clinician researcher, who were not part of the EP. All group discussions were conducted online using a video conferencing platform [16].

Study setting

The study was done as part of the work of the Collaboration for Research, Implementation and Training in Critical Care in Asia- Africa (CRIT Care Asia-Africa—CCAA). Established in 2019, this community of practice supports a network of nationally led ICU registries within nine countries in Asia and eight countries in Africa, representing 260 + acute and critical care departments. A detailed description of the project is published [17]. Stakeholders representing acute and critical care services from all CCAA collaborating countries were invited to participate. In addition to the members of the CCAA, researchers with extensive expertise in ICU registries, and or quality indicators and health care evaluation were invited also from other parts of the world including the Americas, Europe, and Oceania.

Study participants

Forty-seven individuals were invited to form the EP, representing clinicians primarily involved in ICU care (physicians, nurses, and allied health professionals), physicians from other specialties related to critical care, researchers, and patients who have survived ICU care or patient representatives. These EP members were identified as active members of the current CCAA registry and wider research network. For the third round of voting, the EP, were asked to invite 1–3 additional stakeholders from their respective national networks in CCAA participating countries. The EP were asked to include representatives from health care with similar experience as the EP (Electronic supplementary material 1).

Data collection and analysis

Candidate indicator list identification

A candidate list of indicators was compiled by the AC from three sources; those identified through a published scoping review [18], those already used in national registries collaborating with CCAA [17], and those identified through a stakeholder prioritisation exercise at a national CCAA meeting. Additionally, participants were able to add indicators in the first round of voting. Indicators were categorised using the Lancet Global Health High-Quality Health Systems framework’s three domains: foundation, care process, or quality impact [11]. Each indicator was described, defined (based on existing published definitions), and graded for evidence prior to presenting to the EP for voting [19]. Where indicators had multiple published definitions, participants were asked both to prioritise the indicator and the most appropriate definition.

Scoring of indicators

The Delphi had four rounds (Fig. 1). In rounds one and two, respondents appraised each indicator based on two criteria: validity, and feasibility (Fig. 1). Consensus for validity and feasibility was considered a prerequisite to assessing sensitivity to change and reliability. In round three, candidate indicators were assessed for validity, feasibility, sensitivity to change and reliability. In this round, voters were additionally asked to report perceived barriers and facilitators to implementation. Perceived barriers and facilitators were then categorised according to the five domains of the Consolidated Framework for Implementation Research (CFIR); intervention characteristics, outer setting, inner setting, individual characteristics, and processes. CFIR was chosen for its systematic and replicable identification of constructs which may promote or inhibit implementation of interventions in clinical practice [20]. Finally, in round four, the indicators were prioritised based on their ability to influence five priority impacts—cost of care to recipient and provider, staff wellbeing, patient safety, and patient centred outcomes [21]. The EP prioritised each indicator on all five criteria as being low, medium or high impact.

Fig. 1
figure 1

Delphi process and criteria for voting and prioritisation

Full size image

An indicator was entered into subsequent rounds of voting if it achieved a median score of ≥ 4 using a 5-point Likert scale (strongly disagree = 1 to strongly agree = 5), and if there was consensus across the voting members (“consensus to retain”). Consensus was defined as an interquartile range (IQR, defined as quartile 3-quartile 1) of < 1 for validity and feasibility, and an IQR of ≤ 2 for reliability and sensitivity to change. The criteria for reliability and sensitivity are less stringent because these characteristics may differ in interpretation across health systems, whereas feasibility and validity should be uniform. Indicators scoring a consensus median of < 4 were removed from further rounds (“consensus to remove”). Indicators which achieved a median score of ≥ 4 or < 4, but where there was no consensus were put forward to the next round of voting (“no consensus to retain or remove”). At the end of each voting round, The EP convened to discuss the results, with a view to moving towards consensus in subsequent rounds. Discussions focussed on those indicators where there was absence of consensus, variation in definition, and or divergence in voting patterns. [19, 22]. To inform these discussions voting was described based upon geographical distribution of respondents. The regions considered were based on the United Nations division of the world’s geographical regions—Asia, Africa, Americas, Europe, and Oceania [23]. The consensus process (rounds one to three) was intended to provide a list of stakeholder selected indicators, which having already achieved agreement across the four criteria, could then be prioritised in the final round of voting (round four). In case of consensus not being achieved, an alternative strategy was included in the protocol, whereby following round three, indicators that received a Likert of > 3 from more than 50% of respondents were retained for round four, similar to previous Delphi studies [10, 24]. In addition participants were provided with a free text box after every section in the survey (rounds one and three) where they were asked to comment if they had any concerns with the definitions of the indicators. The definitions that were contested in this manner were then reviewed for accuracy and feasibility of collecting data, and where necessary, alternatives were proposed by the DWG.

Results

Study participation

The study was conducted between March 2021 and October 2021 (Fig. 2). Of the 47 EP members invited, 43 agreed to participate. For round three, an additional 70 stakeholders were invited from the EP’S respective networks and 31 responded. A total of 71 clinicians, researchers and healthcare professionals representing 30 countries participated (Electronic supplementary material 2). Response rates were 50% or greater for each round. Fifty one participants (72%) completed all four rounds. The numbers, demographics, and professional characteristics of participants at each round are described in Table 1. Specialities of critical care, surgery, internal medicine and public health were represented in each of the four rounds (Table 1). Patient and patient representatives invited to the EP declined, citing lack of experience in critical care internationally, and or unfamiliarity of quality-of-care metrics (Table 2).

Fig. 2
figure 2

Overview of the Delphi process. *DWG—Definitions working group. **These indicators were removed during voting due to non-consensus, but were re-introduced at the request of the EP

Full size image
Table 1 Respondent characteristics
Full size table
Table 2 Prioritised indicators for piloting in the CCAA collaborating ICU registries
Full size table

Indicator selection

Eighty-four candidate indicators constituted the initial list put forward to round one. Sixty-seven indicators had one definition, whereas 17 indicators had multiple published definitions (Electronic supplementary material 3). Following three rounds of voting, 70 indicators and 70 definitions were retained (20 due to an absence of consensus) (Fig. 2). Given the lack of consensus, we agreed to proceed with our a priori plan to include indicators receiving a Likert of > 3 from more than 50% of respondents into round four. This further removed 21 indicators, leaving a total of 49 indicators considered valid, feasible, sensitive to change and reliable. Variation in voting patterns across geographic regions is described in Electronic supplementary material 6. Variations in voting revealed inequalities in access to laboratory, point of care diagnostics and equipment, which hindered feasibility and reliability of metric reliant on this information for measurement. Appraisal of the remaining indicators, it was noted that no patient reported outcomes were retained, due to low scoring regarding feasibility of data collection. EP discussion and review of voting patterns revealed these indicators scored highly for validity, and therefore the EP concluded that these indicators should be considered for prioritisation. The EP requested that the eight quality impact indicators measuring patient centred and medium term outcomes should be included in round four. Prioritisation of the final 57 indicators (from round three–16 foundation, 27 care processes and 14 quality impacts) using the five priority setting categories resulted in 14 indicators being prioritised; (4 foundation measures, 6 process measures and 4 quality impacts) (Electronic supplementary material 4).

Barriers and facilitators

Eight discrete barriers and seven facilitators to implementation were identified, and categorised into the four CFIR constructs: intervention characteristics, inner setting, outer setting, and processes (Fig. 3). The most frequently perceived facilitator to implementation described by respondents was the use of electronic data collection through the use existing clinical quality registries to routinise and standardise data collection (n = 54/57, 95%) The burden of data collection (n = 53/57, 93%), and an existing lack of uniformity for the procedure (n = 36/57, 63%) were identified as barriers to implementation. Similarly, co-implementation of indicators as part of a cycle of audit and improvement alongside data pertaining to case mix and outcomes was described as an important facilitator to both implementation and impact (n = 41, 72%), as was the opportunity to participate in quality improvement programmes (n = 25/57, 44%). The absence of standardised definitions and associated expertise in interpretation of measurements (n = 45/57, 79%) was seen as a further barrier to implementation and reinforced findings of the narrative analysis of the definitions working group described above (Electronic supplementary material 5).

Fig. 3
figure 3

Barriers and facilitators to implementation. Constructs (CFIR): C1—Intervention characteristics; C2—Outer setting; C3—Inner setting; C4: Characteristics of individuals; C5: Process

Full size image

Discussion

Our multinational four round Delphi study resulted in the selection of 57 indicators for use in critical care settings in LMICs, and among these 14 were prioritised. Individual ICU networks will in addition have a choice as to which indicators they select for implementation. We anticipate the selection of indicators will reflect context specific factors and priorities for improvement in the different settings. These indicators will be evaluated for feasibility of implementation, and for their ability to reliably measure associated outcomes. This study brings new perspectives to the ongoing discussion [11, 18, 25] regarding indicator selection for benchmarking ICUs internationally, and the potential implications that use of some indicators for pay for performance may be having on efforts to improve quality of care. This study adds to the existing international literature by providing much needed representation from previously under-represented ICU services and critical care populations in low- and middle-income health systems.

Disparities in access to basic resources for delivery of safe critical care were evident and were reflected in the voting. Comparatively low ratios of trained healthcare providers to patients, and inequalities in access to basic resources essential for ICU care (monitoring, infusion pumps and medications, including oxygen as described elsewhere in the literature) was a driver in our study for the measure of their availability being included as a foundation indicator [2, 26, 27]. Healthcare associated infections, pressure related injury and thrombus, were considered important alongside the universal public health primacy of multi-drug resistant infection. Absence of laboratory services and concerns over reliability of sampling techniques and diagnostic stewardship, was a concern for respondents when appraising indicators for feasibility and reliability. The ongoing need to improve the quality of microbiology and diagnostic services particularly in Africa is well described and, despite significant investment, remains a barrier to both quality improvement efforts and clinical trials [28, 29]. Whilst investment in laboratory and diagnostics services (including microbiology and haematology) has increased opportunities for training, ongoing disparities in infrastructure remain and are likely to hamper efforts to improve antimicrobial stewardship and infection control within critical care [28,29,30,31,32]. In addition, absence of or lack of existing peer agreement regarding indicators’ definitions further hampered voters' ability to achieve consensus during selection and raised concerns as to how continual refinement in definitions undermines indicator reliability, prevents accurate benchmarking and opened the door to “gaming” of indicators, particularly in healthcare systems where indicators are associated with pay for performance; common in South and SouthEast Asia, and emerging in some African countries [33,34,35,36].

There was a high proportion of process indicators selected (47.3%) in our study, compared to the 2012 European Society of Intensive Care Medicine (ESICM) task force (where 22.2% were selected [7]). This shift may reflect growing awareness of the potential impact that omissions in daily care processes have on excess morbidity and mortality [31]. It may further reflect the impact that staffing, training and until recently, the relative absence of infrastructure of reliable replicable data driven service evaluation has on patient outcomes and on efforts to improve care delivery. Outcome measures (quality impacts) were similar to those selected by previous consensus studies from Europe and the UK [1, 7], with the addition of cost to both patient and provider, which is congruent with the need for out-of-pocket payments for care in many LMIC settings [11]. Indicators implemented in the network, will be evaluated for feasibility of collection and validated for their association with outcomes. As is common practice with national ICU registries internationally, population outcomes will be adjusted for case-mix, and risk using already validated and internationally comparable prognostic models (APACHE IV, SAPS III, and E-TropICS) [37, 38]. Impact of organisational factors, including team structure, resource availability and culture of quality improvement are also being explored using mixed methods. Definitions used to determine adverse events (for example incidence of healthcare associated infection) have been chosen from published literature and will be assessed for feasibility and reliability in the different ICU populations.

The use of registry-based data collection, the co-design of implementation processes and the use of feedback tools were identified as important strategies to overcome known barriers of feasibility and reliability of data collected [22,23,24,25]. Current measures of medium-long-term functional outcomes, whilst perceived as a priority, were considered poorly reflective of some social and lifestyle constructs for communities in Africa and Asia. For example, social and lifestyle constructs including driving, or playing sport did not reflect respondent experience of patient or family priorities, particularly for elderly or poorer economic quintiles of the population. Alternatives including being able to carry and care for younger family members, or being able to work on the farm, were proposed. The need for greater investment in research exploring patient and public priorities for recovery and quality of life after critical illness remains.

From all representatives including those from the most fragile health systems there was a recognition that the expanding focus of critical from episodic care towards longer term holistic provision of care which requires greater understanding of the impact of critical care on patients, their families and wider population health well beyond the hospital walls. The absence of integrated health and social care systems and lack of interoperability in healthcare data was seen as a significant barrier to measuring the impact of existing ICU services and inform how to invest in services as they rapidly expand in LMICs. Even if solutions such as telephonic follow-up or cohort follow up for specific critical care populations could facilitate data collection of the indicators of quality, concerns remained as to whether there is sufficient expertise within health ministries and healthcare policy making agencies to interpret information.

The perceived importance of these measures in evaluating quality of care, and the feasibility and acceptability for communities of capturing outcomes after ICU is reflected in the request from the EP to include such indicators in the prioritisation exercise (round 4) despite these indicators failing to meet the threshold for inclusion based on consensus scoring. Whilst the tension of overriding the Delphi process is a potential weakness of the study, the ACs decision to include these indicators for prioritisation was intended to strengthen the studies aim of identifying stakeholder prioritised indicators. The unique opportunity of the network’s community of practice is the uplift in infrastructure and methods to be able to overcome potential barriers (such as feasibility of follow up) and to undertake parallel PPIE research to explore the acceptability of follow up services in Asia and Africa, where the network is active. Of the 57 selected indicators selected after round three, 14 indicators were prioritised.

Limitations

The stakeholder representation for this Delphi was dominated by experts working in Asia and Africa, where the CCAA collaborating registries are operational. There was, however, representation from ICU experts (with expertise in and experience of using clinical registries) from South America. It must also be acknowledged that not all countries in Asia and Africa were included and those included have access to an online ICU registry and are part of a research collaboration, thus more likely to be engaged in and aware of the value of quality improvement and research as part of high-quality health systems. With increased focus on quality improvement across LMICs and investment in electronic systems like DHIS2 [39], our findings go beyond just Asia and Africa, and may be applicable to LMICs.”

Despite invitations to all sectors of clinical healthcare teams, nursing and allied healthcare professional representation remained limited and may have contributed to the gender imbalance observed in this Delphi; the majority of respondents were male. The absence of patient representation, despite invitations to patients and patient advocacy experts to participate, is a limitation of this study. Patient representatives approached declined participation citing concerns regarding their understanding of quality in healthcare, and in the case of advocacy experts, their understanding of critical care in the LMIC context. The pandemic brought into focus the absence of understanding of critical care by the general public and indeed other non-acute sectors of healthcare. Public engagement and education regarding the role critical care services may play in improving population health is needed. However, representation from stakeholders in Asia and Africa in this study, often underrepresented in similar research, was proportionally higher than reported in other published studies. [10, 21].

Conclusion

This Delphi study resulted in the selection of 57 indicators (16 foundation, 27 process and 14 quality impact) suitable for use in LMICs. Whilst recognition of the need for ongoing investment in health service infrastructure remained, stakeholders voting patterns demonstrated an increasing awareness and prioritisation of the need to measure care processes within the ICU and its potential to improve care standards and reduce avoidable harm. Despite the challenges of maintaining validity and reliability of existing indicators of ICU outcome when applied to more diverse critical populations and organisational structures, the Delphi process underscored the importance of these indicators. Context specific definitions, addressing known barriers to feasibility of data collection and absent information, were identified. This indicator set may become a tool to support benchmarking and quality improvement programs for ICUs beyond those represented here.