Background

The increasing burden of trauma and non-communicable diseases in low- and middle-income countries (LMICs) has emphasized the need for effective emergency care to alleviate the morbidity and mortality associated with acute illness and injury [1–4]. This has led international organizations, including the World Bank, World Health Organization, and United Nations Children’s Fund, to place substantial emphasis on the development and strengthening of systems of emergency care in resource-limited settings [4–6].

As emergency care in LMICs expands, there is a growing need to measure and improve the quality and safety of this care. Well-developed quality assurance systems currently exist in high-income countries [7, 8], where the development and use of quality indicators has led to major improvements in the standard of emergency care provided [9]. While systematic performance measurement is the foundation of quality health care [10], quality and safety indicators used in developed countries may not be appropriate in resource-limited settings [9, 11]. Indeed, little is known about the metrics being used to measure emergency care in LMICs, and to our knowledge, no study has cataloged which metrics are being used. This limits the ability of emergency departments (EDs) in low-resource settings to implement quality assurance programs.

The objective of this systematic review is to catalog and classify existing performance metrics that have been used to measure the quality of ED care in resource-limited settings.

Methods

Search strategy

A medical librarian searched MEDLINE, Embase, and CINAHL from the earliest available date to September 30, 2013. The following search terms were used: quality, quality assurance, quality indicators, utilization review, combined with any of the following: emergency, emergency medical services, emergency service, accident and emergency, emergency department, and any of the following: developing countries, third world, and resource limited (or low or poor). The search was restricted to English language articles.

The gray literature was searched through an internet-based search of the websites of relevant international and emergency medicine organizations such as the World Bank, International Federation for Emergency Medicine, International Medical Corps and United Nations, as well as a Google search using combinations of the following terms: quality, quality assurance, quality indicators (or measures), performance indicators (or measures), safety indicators (or measures), combined with any of the following: emergency, emergency medicine, emergency medical care, emergency services, emergency unit (or department), and any of the following: developing, resource-limited, and low- or middle-income countries.

Article selection

Studies were eligible for inclusion if they were conducted in a low- or middle-income country, as defined by the World Bank classification system [12], and addressed quality markers, indicators, or metrics for care in an ED or emergency unit. Studies conducted in multiple countries were included if one or more of the countries was a LMIC. A metric was defined as a performance measure that assessed a predefined quality standard. If an article analyzed the quality of care in the hospital as a whole, including the ED, the article was included only if the article separately reported metrics measured in the ED. Studies of prehospital care, emergency obstetrics, and secondary injury prevention were excluded. If a study included both prehospital and in-hospital metrics, it was included only if the in-hospital metrics were separately listed.

Articles were excluded if they were opinions or review articles that did not feature original data. Articles that described potential indicators, but did not implement them or measure them, were also excluded, as the focus of the present review was on indicators that have been previously utilized.

One author (ELA) initially reviewed the titles and abstracts of all articles identified by the search terms to exclude all clearly ineligible articles. The remaining titles and abstracts were re-reviewed by two authors (ELA and SAR), and a consensus was reached to create a list of potentially relevant articles. The full text articles were then reviewed by three authors (ELA, SAR, and RHM) to confirm eligibility. Given the limited literature on this topic, articles were not excluded based on quality of the study or publication.

Data extraction and analysis

Three authors (ELA, SAR, and RHM) reviewed the full text of all relevant articles using a standardized form to extract individual quality metrics and study details. If a study examined both ED and hospital care, only the ED metrics were included. However, if a study looked exclusively at care within the ED, but included metrics or outcomes that occurred after the ED stay, such as mortality, these metrics were included.

For ease of comparison, certain structural metrics were collapsed into predefined categories. For example, metrics examining availability of specific medications were combined into a single metric by predefined medication class.

Once extracted, each metric was categorized by a predefined matrix based on the IOM framework of healthcare quality (Table 1) [13]. These were then further classified into the Donabedian framework of health care consisting of causally linked and measurable categories (Table 2) [14].

Table 1 The Institute of Medicine framework of healthcare quality
Full size table
Table 2 Donabedian framework of health care
Full size table

Each quality metric was assigned to only one domain. A fourth author (JDS) then independently reviewed and classified the extracted quality metrics, with a weighted kappa of 0.89.

Articles applying the WHO/IATSIC guidelines

Several studies used the indicators in the World Health Organization (WHO) and International Association for Trauma and Intensive Care (IATSIC) Guidelines for Essential Trauma Care. These guidelines offer a toolkit of over 200 metrics for the internal assessment of trauma care at the hospital level, focused on human resources (staffing and training) and physical resources (infrastructure, equipment, and supplies). While the guidelines reference emergency care, they are intended to assess specifically trauma capacity of a hospital as a whole, which was not the present focus of this review. A number of studies have been performed applying the WHO/IATSIC indicators, and thus including them in the primary analysis would have disproportionately weighted the indicators of this study. Therefore, quality indicators found in this group of articles were examined separately. Only reported metrics from each study were extracted.

Results

The literature search identified 1705 titles (Fig. 1). Of these, 97 were eligible for full text review. Two articles could not be located, after exhaustive search by a trained medical librarian. Of the 95 reviewed, 30 met inclusion criteria. The references of included articles were also reviewed, yielding an additional 4 articles for inclusion. In total, 34 articles were included, 6 of which reported the implementation of the WHO/IATSIC guidelines [15–49]. The summary characteristics of the non-WHO/IATSIC included articles are listed in Table 3. Detailed descriptions of each article included are listed in the Additional file 1.

Fig. 1
figure 1

Process of inclusion of studies in the systematic literature review

Full size image
Table 3 Summary characteristics of articles (each country is counted individually in multinational studies)
Full size table

Excluding the WHO/IATSIC articles that were analyzed separately, 180 quality metrics were extracted from the remaining 28 articles, including 129 unique indicators. The majority of all reported measures were not disease-specific (n = 126; 70 %) but focused on metrics that applied to patients with a variety of diseases. The 54 measures that were disease-specific focused on illnesses related to the following: respiratory (n = 23), systemic states (n = 9), hematologic (n = 7), circulatory and cardiovascular (n = 6), neurologic (n = 4), trauma (n = 2), endocrine, metabolic and nutritional disease (n = 1), fluid and electrolyte disorders (n  = 1), and gastrointestinal diseases (n  = 1).

Most metrics were process (n = 102; 57 %) and structure measures (n = 49; 27 %). Only 16 % (n = 29) were related to outcomes (Table 4). Regarding the IOM domains, most metrics evaluated the effectiveness of care (n = 94; 52 %). These were predominantly markers of effective processes, such as adherence to a full physical exam and appropriate test ordering, or of effective structures such as the availability of essential supplies. A small number dealt with the effectiveness of outcomes, such as mortality. Metrics assessing timeliness of care (n = 51; 28 %) dealt primarily with processes such as time to provider and outcomes such as length of stay. Few metrics addressed patient-centered care (n = 20; 11 %); those that did looked primarily at patient satisfaction. Seven percent (n = 14) of metrics addressed the safety of care and focused on complications of care and the appropriate use of medications. Resource-efficient (n = 5; 3 %) and equitable (n = 2; 1 %) measures were rare.

Table 4 Frequency of indicators extracted from non-WHO/IATSIC studies, classified by Donabedian and Institute of Medicine domains
Full size table

Among the articles that implemented the WHO/IATSIC surveys, 336 metrics were extracted. Many of these were repeated within articles; a total of 153 unique metrics were identified. The majority of the metrics were related to the effectiveness of the structure of care (n = 141; 92 %). The remaining metrics dealt with safe structure (n = 7; 5 %), safe process (n = 4; 2 %), and efficient process (n = 1; 1 %).

Discussion

The EM quality literature provides strong evidence that quality improvement programs can improve quality of care and patient outcomes [7, 8, 49]. Understanding that improvement requires measurement [7, 8], the availability of applicable measures for emergency care in LMICs is essential. While there has been dramatic growth in the delivery of emergency care in LMICs over the last decade, little is known about the quality of care or how to evaluate it.

Through a rigorous search strategy and structured data extraction, this systematic review collected and analyzed published ED quality metrics in LMICs. Our data show that only a limited number of metrics have been reported, the majority of which focus on structures or processes of care, rather than on patient outcomes. The limited metrics suggest a pressing need to develop and implement performance measures that reflect the spectrum of emergency care in LMICs.

Our study shows that when applying a structured framework for quality metrics to the over 150 metrics currently used to measure the quality of emergency care in LMICs, these metrics do not achieve balance. The majority of these metrics are focused on process and structure, likely reflecting the greater availability of data in these domains. Process metrics, making up over half of all metrics reported in our study, were predominantly centered on operational measures looking at the effectiveness and timeliness of ED processes. Although literature in high-income countries suggests that the most successful performance measures for quality improvement are outcome metrics related to ED time intervals (length of stay, arrival to assessment/admission) and patient centeredness (72-h ED returns, patients who left without being seen) [8, 50, 51], we found that only 16 % of metrics in LMICS were outcome based. Even fewer (11 %) were patient-centered. Future work is needed to analyze why patient-centered and outcome indicators have not yet been implemented and to develop contextually appropriate measures.

It is interesting to note that few resource-efficient metrics were reported. EDs in LMICs are increasingly facing the pressures of high patient volumes, limited resources, and an ever-high burden of disease [52]. Given this landscape, it is essential that the care being delivered is resource-efficient and high quality. Unlike in high-income settings, where individual providers typically do not face resource constraints, in a low-resource setting, the use of an expensive medication or lab test on one patient may consume the resources needed to treat the next. The limited resources and significant unmet need for health care in LMICs make it essential that care is both efficient and equitable. The ability to provide more with less is inherently tied to the ability to create streamlined processes and efficiency in both operations and supply-chain management. The WHO has noted that health systems in low-income countries have the greatest potential for increasing efficiency with minimal investment [53, 54]. Our study suggests the need to identify metrics that can measure these efficiencies and contribute to their improvement.

Prior efforts have been made to identify feasible quality metrics in emergency care. In addition to metrics which came out of the International Federation of Emergency Medicine’s Symposium for Quality and Safety, an expert consensus study conducted in South Africa identified 58 performance indicators that they deemed to be feasible to measure in low-resource settings (37 structure based, 20 process based, and 1 outcome based) [52, 55]. Our study shows that few of these are being reported currently. Only 20 of the proposed measures (34 %) were identified in our review (11 structural measures, 4 process measures in the non-WHO/IATSIC studies, and an additional 5 structural measures in the WHO/IATSIC studies). This suggests that there are a large number of potentially feasible quality indicators that are not being studied in LMICs. Future research should examine the reasons for this discrepancy and either modify these metrics or support their implementation as needed.

While the metrics identified in this review have all been successfully measured in LMICs, the ease of measurement is not compared or documented. The well-documented barriers that exist to measure these metrics in developed countries are likely more significant in LMICs. Lack of senior management with strong commitment or training in quality improvement methods, limited resources to collect and analyze data, and a lack of clarity around which metrics are most important have all been noted to limit institutional abilities to effectively measure the quality of care [7]. These challenges are magnified in LMICs, particularly within EDs, as emergency medicine is in its nascency in many countries.

Limitations

The study was limited to English language articles and may have missed metrics reported in the non-English language literature. Although our search terms were broad, there may be articles using different terms that we did not capture. Similarly, it will not capture unpublished metrics currently in use. There are no standardized definitions for the classification of metrics in the IOM and Donabedian domains, resulting in a degree of subjectivity in their classification. To address this, three authors reached consensus on each metric, and a fourth blinded reviewer re-classified the metrics with a kappa that was robust. Finally, the IOM and Donabedian domains were developed for high-resource settings, and their applicability to a lower-resource setting is unknown.

Conclusions

As emergency medicine continues to grow as a field in LMICs, there is an increasing need for effective metrics to measure the quality of this care. This systematic review of performance measures suggests that although there are a number of published quality metrics currently used to assess emergency care in LMICs, these do not adequately assess all aspects of emergency care. This study has demonstrated that broad metrics have been applied in LMICs, however identified the need for the development of more comprehensive measures that are locally applicable. As metrics for LMICs are developed, they must be implemented and then reported on to develop global standards of quality measurement in emergency care.