Siginificance

The Adverse Outcome Index and its severity-weighted measures, the Severity Index and Weighted Adverse Outcome Score, are perinatal health indicators that were developed to monitor the safety and quality of care during labour and delivery. This study established the feasibility of operationalizing these indicators at the population level using routinely-collected data in a provincial perinatal database, providing a new tool for population-based surveillance of obstetrical care during labour and delivery. The wide variation observed between institutions in risk-adjusted rates of the Adverse Outcome Index, Severity Index and Weighted Adverse Outcome Score suggests that room for improvement in clinical practices may exist at some sites.

Introduction

Obstetrical care has conventionally been monitored using indicators such as preterm birth, maternal or neonatal death, and cesarean delivery [1]. However, the relevance of these indicators for assessing the quality and safety of labour and delivery is questionable [2]. Mortality is very rare and does not capture non-fatal adverse events, while recording the proportion of births delivered by cesarean does not reflect whether the surgery was appropriate or not [2]. More recent indicators such as the percent of elective deliveries at early term gestation (37–38 weeks) developed by the Joint Commission [3] have been shown to correlate poorly with maternal and newborn health status [4]. Comprehensive measures that combine a wide range of adverse maternal or neonatal events into a single composite index have been proposed as a better approach for summarizing obstetrical care [5], but typically do not account for the relative severity of different adverse events.

The Adverse Outcome Index (AOI), Weighted Adverse Outcome Score (WAOS), and Severity Index (SI) were developed to overcome the limitations of existing quality and safety indicators for obstetrical care [6]. The AOI is a composite of 10 adverse maternal or neonatal events related to labour and delivery: maternal death, intrapartum or neonatal death, uterine rupture, maternal admission to the intensive care unit (ICU), birth trauma, unanticipated operative procedures, neonatal intensive care unit (NICU) admission for greater than 24 h, 5-min Apgar score <7, blood transfusion, and 3rd or 4th degree perineal tear. The WAOS is based on the same components but weights the events according to their severity (i.e. combines frequency with severity), while the SI describes the average degree of severity among pregnancies with an adverse event. These indicators were derived through a multi-stage process that involved reviewing the literature for existing quality measures, convening a consensus conference of leaders in obstetrics, nursing, and anaesthesia in the United States to review the existing measures and identify additional candidates, and finalizing the indicators’ definitions in a second consensus conference [6].

The AOI and its severity-adjusted measures have been used to assess the effectiveness of interventions at the hospital-level [7], but their value in population-based monitoring of health care delivery and patient safety has not been examined. In this study, our goals were to establish the feasibility of implementing the AOI, WAOS, and SI in British Columbia using routinely collected data available in our provincial perinatal database and to examine variation over time and between hospitals in the AOI, WAOS and SI.

Methods

Our study population included all singleton births in British Columbia, Canada, between April 1st, 2004 and December 31st, 2013. Obstetrical, demographic, medical, and neonatal data on these deliveries were obtained from the BC Perinatal Data Registry (BCPDR), a quality-controlled population-based database that contains abstracted medical records from over 99 % of live births and stillbirth ≥20 weeks or ≥500 g in the province (including home births) [8]. The BCPDR is maintained by Perinatal Services BC, a public provincial agency with a mandate to enhance the delivery and quality of perinatal health care services in the province. Maternal and neonatal medical records are abstracted by Health Information Management professionals, who are trained through a minimum 2-year education program followed by a national certification. Data quality is maintained by use of provincially-standardized medical chart forms, use of a specialized data entry tool containing logic checks, and ongoing quality-assurance reviews at the hospital and provincial levels. The BCPDR has been used extensively for surveillance and research purposes [9–14] Pregnancy terminations were excluded. Ethics approval was obtained from the Research Ethics Board of the University of British Columbia/BC Children’s and Women’s Hospital (#H14-00990).

The AOI was calculated as the percent of deliveries in which an adverse labour and delivery outcome occurred to either mother or infant. An adverse outcome was defined as the occurrence of any of: maternal death during delivery admission, intrapartum stillbirth or in-hospital newborn death ≥2500 g at or beyond 37 weeks’ gestation with no congenital anomalies or fetal hydrops, uterine rupture, maternal admission to ICU (based on use of maternal intensive care procedures such as intubation or ventilation as a proxy), birth injury, unanticipated operative procedure, neonatal care unit admission (NICU) ≥2 days or transfer within 24 h of birth to a facility with a NICU for a non-anomalous infant ≥2500 g at or beyond 37 weeks with no fetal hydrops, 5-min Apgar score <7 for a non-anomalous infant ≥2500 g at or beyond 37 weeks with no fetal hydrops, blood transfusion, or 3rd or 4th degree perineal tear. The NICU admission time required to qualify as an adverse outcome was increased to ≥2 days from the original definition of 24 hours as this was believed to better reflect clinical practice patterns in British Columbia (i.e. to ensure that newborns admitted to the NICU for observation only were not classified as having an adverse outcome). The specific BCPDR variable, International Classification of Diseases Tenth Revision, Canada, and Canadian Classification of Health Interventions codes used to define each outcome are detailed in the Appendix.

The WAOS was calculated by first assigning points to each adverse outcome (i.e. weighting the outcomes according to severity), summing the total number of points in all deliveries with an adverse outcome, then dividing the total number of points by the total number of deliveries. As established in the original WAOS development process [6], a score of 750 was assigned for maternal death, 400 for intrapartum or in-hospital newborn death ≥2500 g, 100 for uterine rupture, 65 for maternal ICU admission, 60 for birth injury, 40 for unanticipated operative procedure, 35 for NICU admission ≥2 days or transfer within 24 h of birth to a facility with a NICU for an infant ≥2500 g, 25 for a 5-min Apgar score <7, 20 for blood transfusion, and 5 for 3rd or 4th degree perineal tear. The SI was calculated as the total number of points assigned for each adverse event divided by the total number of deliveries with an adverse event (i.e. the average severity of adverse events among those deliveries experiencing an adverse event). Thus, the WAOS reflects a combination of the frequency and severity of events, while the SI examines how severe the adverse events were (within the pregnancies that experienced an adverse event). The SI and WAOS can most easily be interpreted when used for comparisons: monitoring changes in the measures over time (i.e., flagging increases or decreases in more severe events) and contrasting severity of events between hospitals, regions, or care providers.

We calculated the risk of AOI, WAOS, and SI for each year and for each hospital with an average obstetrical volume of >10 deliveries per year with pointwise 95 % confidence intervals. Hospitals with 10 or fewer deliveries were excluded to avoid inclusion of hospitals that did not have planned obstetrical services (but may perform a small number of emergency deliveries). British Columbia has a universal health care system, and all hospitals are publicly funded. Risks between years or between hospitals were considered to be significantly different if the 95 % confidence intervals were mutually exclusive. Confidence intervals for WAOS and SI were calculated using bootstrapping with 1000 iterations. We ranked each hospital according to risk of AOI, WAOS, and SI, and examined the extent to which an institution’s risk of AOI agreed with the institution’s WAOS or SI by calculating a Pearson’s correlation coefficient.

To account for differences in patient characteristics between hospitals that might influence the apparent AOI risk, we used the obstetrical risk-adjustment approach developed by Bailit et al. [15]. In this approach, a logistic regression model was built to estimate each woman’s AOI risk based on her individual characteristics. This model included all patient factors that were associated with a composite adverse neonatal outcome, severe perineal laceration, or post-partum haemorrhage in the models of Bailit et al., but excluded insurance status (as this is not applicable in the single-payer Canadian health care system) and cocaine or methamphetamine use (due to lack of reliable data). These variables were: maternal age, pre-pregnancy body mass index, smoking during pregnancy, obstetrical history (parity combined with previous cesarean), premature rupture of membranes, multiple births, diabetes in pregnancy, hypertensive disorders of pregnancy, placenta previa, placenta accreta, placental abruption, and birthweight for gestational age percentile. The decision on which variables to include in the model was made a priori, and we did not exclude any variables based on lack of statistical significance. A robust cluster variance estimator was used to account for clustering of women within hospitals. As previously described, [15] we then averaged the predicted risks for all women within the same hospital to obtain the expected adverse outcome rate for each hospital (i.e., the hospital’s expected rate of adverse outcomes, given the characteristics of the women delivering there). The actual AOI risk was compared with the expected risk by calculating an observed to expected ratio. A risk-adjusted rate was then obtained for each hospital by multiplying its observed-to-expected ratios by the population AOI rate with bootstrapped 95 % confidence intervals. We were unable to apply a risk adjustment methodology to the hospital-specific WAOS and SI as these measures have non-standard statistical distributions (e.g., WAOS is highly skewed with a majority of 0 values) that are not amenable to multivariable regression modelling (i.e., by estimating a mean, median, or odds).

We conducted a number of sensitivity analyses to assess the robustness of the measures to alternative definitions. First, we included dehiscence or hysterectomy in the uterine rupture component. Second, we restricted the blood transfusion component to cases in which a post-partum haemorrhage was also coded. Third, we added visceral injury to the list of unanticipated operative procedures. Fourth, we included stillbirths with ‘unknown’ timing to the intrapartum stillbirths, and lastly, we excluded 3rd and 4th degree tears from the Index. We additionally examined the potential impact of under-capture of maternal deaths and ICU admissions by tripling the incidence of each, and recalculating the AOI, WAOS, and SI.

Results

There were 410,054 singleton births in British Columbia between 2004 and 2013. The average maternal age was 30.2 years (±5.6 SD), 204,263 women (46.8 %) were nulliparous, and the average pre-pregnancy body mass index (BMI) was 24.2 kg/m [2] (±5.2 SD) of the 304, 920 (70 %) women with available pre-pregnancy BMI.

Of all deliveries, 5.7 % experienced one or more components of the AOI. Table 1 shows the occurrence of each component of the AOI. The most common component of the index was 3rd or 4th perineal tear (29.9 per 1000 deliveries), while low 5-min Apgar score and NICU admission were the second and third most common (11.5 and 6.5 per 1000, respectively). Accordingly, maternal components of the AOI were more common than the neonatal components (40.3 vs. 18.8 per 1000 deliveries, respectively).

Table 1 Adverse Outcome Index in 410,054 deliveries in British Columbia, 2004–2013
Full size table

Figure 1 shows the trends over time in rates of the AOI. The AOI did not differ significantly from year to year, with a risk of 5.6 per 100 in 2004 (95 % CI 5.3–5.8), 5.9 per 100 in 2007 (95 % CI 7.3–7.8) and 5.7 per 100 in 2013 (95 % CI 5.5–5.9). The maternal components of the AOI also did not differ significantly between years during the time period [e.g., risk of 3.8 per 100 (95 % CI 3.6–4.0) in 2004 vs. 4.1 per 100 (95 % CI 3.9–4.3) in 2013]. In contrast, the neonatal components decreased post-2007 [2.2 (95 % CI 2.1–2.4) in 2007 vs 1.8 (95 % CI 1.7–2.0) in 2008 and 1.9 (95 % CI 1.7–2.0) in 2013], which upon closer examination was revealed to be caused by NICU admissions. Temporal trends in the WAOS and SI are plotted in Fig. 2. The WAOS was not significantly different between years, while the SI decreased during the study period from 29.3 (95 % CI 26.7–31.9) in 2004 to 23.9 (95 % CI 22.0–25.8) in 2013, suggesting that the severity of the adverse outcomes that occurred lessened over time.

Fig. 1
figure 1

Temporal trends in the AOI in British Columbia, 2004–2013

Full size image
Fig. 2
figure 2

Temporal trends in the WAOS and SI in British Columbia, 2004–2013

Full size image

The crude risk of AOI among the 52 hospital ranged from 2.8 to 9.6 %, with a median of 5.4 % [Interquartile Range (IQR) 4.9 to 6.2]. As seen in Fig. 3, a number of institutions had rates significantly below (n = 8) or above (n = 15) the provincial median rate (i.e., 95 % confidence intervals did not cross the provincial median rate). Adjustment for patient case-mix had a modest effect on the institutional rates, with most pronounced differences between crude and adjusted seen for the institution with the highest crude rates. There were 8 institutions with adjusted risks significantly below and 18 with adjusted risks significantly above the provincial median.

Fig. 3
figure 3

Hospital-level risks of AOI in British Columbia, 2004–2013. Vertical line indicates provincial median risk of AOI

Full size image

Figures 4 and 5 show the inter-institutional variation in WAOS and SI. The median institutional WAOS was 1.62 (IQR 1.28–1.90), while the median institutional SI was 27.0 (IQR 22.6–32.6). As with the AOI, there were a number of institutions with WAOS or SI significantly above or below the provincial median (n = 6 above and 13 below for WAOS and n = 4 above and 13 below for SI). However, institutional rankings according to SI and WAOS differed considerably from those established based on AOI. As shown in Figs. 6 and 7, there was only a modest correlation between risk of AOI and WAOS (r = 0.61) and no association between AOI and SI (r = 0.02), confirming that the AOI and the SI assess distinct aspects of labour and delivery safety.

Fig. 4
figure 4

Hospital-specific risks of WAOS in British Columbia, 2004–2013. Hospitals are ranked according their risk of AOI, and the vertical line indicates provincial median WAOS

Full size image
Fig. 5
figure 5

Hospital-level risks of SI in British Columbia, 2004–2013. Vertical line indicates provincial median SI

Full size image
Fig. 6
figure 6

Association between institutional risk of AOI and WAOS in British Columbia, 2004–2013

Full size image
Fig. 7
figure 7

Association between institutional risk of AOI and SI in British Columbia, 2004–2013

Full size image

Sensitivity analyses examining alternative definitions of AOI components confirmed the robustness of the AOI, WAOS, and SI. Inclusion of dehiscence or hysterectomy in the uterine rupture component, restriction of the blood transfusion component to cases in which a post-partum haemorrhage was also coded, addition of visceral injury to the list of unanticipated operative procedures, and inclusion of stillbirths with ‘unknown’ timing to the intrapartum stillbirths resulted in AOI of 5.6, 5.9, 5.8, and 5.8, respectively, WAOS of 1.54, 1.67, 1.67, and 1.59, respectively, and SI of 27.5, 28.5, 28.9, and 27.4, respectively (vs. 5.8 per 100, 1.6 and 27.4). However, removal of a component, 3rd or 4th degree tears, had a major impact on results: AOI was 3.0, WAOS was 1.4, and SI was 48.8. Tripling the incidence of maternal deaths and ICU admissions had minimal impact on AOI and WAOS (5.8 and 1.7, respectively), with a modest impact on SI (29.3).

Discussion

In this study, we demonstrated that the AOI and its severity-weighted measures, the Weighted AOI and SI can successfully be operationalized in a population-based provincial perinatal database using a combination of diagnosis, procedure, and perinatal database variables. When examining trends over time as well as inter-institutional differences, the results obtained from adverse event severity-weighted indices differed from the unweighted results, supporting the importance of moving beyond mere rates of adverse events (all equally weighted) for understanding safety and quality of obstetrical care.

The risks of AOI, WAOS, and SI in our population were comparable to those previously reported from hospital-based cohorts. In a cluster randomized trial examining the effects of teamwork training on safety of labour and delivery, the average risk of the AOI was 7.2 % (range 4.1–16.5 %) and 8.3 % (range 4.7–12.6 %) in the control and intervention arms, respectively (vs. an institutional median of 6.7 %, range 1.0–12.7 % in our study). The WAOSs were 2.3 and 2.7, and the Severity Indices were 30.6 and 31.9, respectively (vs. 1.9 and 27.1 in our study) [7]. A single center study from the University of Washington Medical Center reported an AOI, Weighted AOI, and SI of 8.6 %, 2.42, and 26.0, respectively [16]. Our stricter definition of NICU admission (admission ≥2 days rather than >24 h) likely reduced our AOI risk compared with previous cohorts. The BCPDR NICU variable also changed in 2008–2010 from measuring bed utilization (i.e., number of infants staying in NICU beds) to level of care needed (i.e., number of infants requiring NICU level of care), which may have further affected the observed rates.

As with any composite outcome, an understanding of the relative frequency of each component of the AOI is critical for proper interpretation [17]. Maternal 3rd or 4th degree tear was by far the most common outcome (29.9 per 1000), accounting for over 50 % of all cases of adverse outcomes. Neonatal outcomes of low 5-min Apgar score and NICU admission ≥2 days were the second and third most common components, at 11.5 and 6.5 per 1000, respectively. As expected, outcomes such as maternal death, perinatal death, uterine rupture, use of maternal intensive care procedures, and birth trauma were very rare. Trends or comparisons using only the (unweighted) AOI will therefore be highly influenced by the proportion of attempted vaginal deliveries, as 3rd or 4th degree tears will not occur in a planned Cesarean delivery. At the same time, the weights used in the WAOS and SI reflect the opinions of experts that may not accurately depict the weights laboring women would attribute to these complications of labour. As with all surveillance measures, the AOI may benefit from further refinement by adjusting weights assigned to each component, altering the weighing of subcomponents (e.g., higher weights to larger volume of blood transfused), or adding/removing components. Understanding how best to assign weights to twin and higher order multiple pregnancies also requires refinement. However, in order to retain the ability to compare rates between jurisdictions, such changes are likely best made based on larger consensus rather than tailoring by individual regions or surveillance groups.

The comparison of institutions within a region (or within obstetrical-volume peer groups) provides a wide angle lens with which to identify sites that appear to have significantly higher or lower rates than average. Although in-depth reviews of the hospitals identified as having higher than average rates in our cohort is beyond the scope of this report, the following general steps should be considered: first, each institution should review its own results for face validity (do they seem reasonable?) If they do not, a chart audit to establish accuracy of provider documentation and identify potential abstraction or coding errors should be conducted. After confirming the accuracy of the data, the frequency of individual components of the index should be examined to identify which factor(s) appear to be driving the higher rate. Best practice guidelines should be reviewed for each factor to review potential areas for improvement. For example, higher uterine rupture rates might prompt a review of institutional protocols for augmentation with oxytocin, while higher transfusion rates might prompt the implementation a haemorrhage toolkit [18] to support the timely recognition of blood loss and early response. Finally, the policies and practices of hospitals with lower than average rates should also be examined to learn from them (what types of training or procedures do they have to handle crisis situations? What types of care models do they use? How is teamwork and communication supported?). Tracking the variation in institutional rates at the regional level will then help to monitor the success or gaps in such efforts.

There was considerable inter-institutional variation in the AOI, Weighted AOI and SI. Although data collection for the BC Perinatal Data Registry is standardized through the use of uniform medical record forms, training of data abstractors using common abstraction instructions and reference manuals, and plausibility checks in the data entry programs, it is possible that some of the between-site differences may be caused by differences in data abstraction rather than differences in adverse events. Likewise, inter-institutional differences in components such as NICU admission could reflect differences in institutional policies (e.g., admission or discharge requirements) rather than true differences in adverse events. Nevertheless, the between-site comparisons in the AOI, WAOS, and SI provide a basis from which to pursue a more detailed investigation into the reasons for the apparently higher (or lower) rates at specific sites, giving the AOI, WAOS, and SI potential value as quality surveillance tools.

Several additional limitations of the study should be noted. Maternal deaths were likely underreported in the BCPDR maternal death variable as it does not capture deaths occurring after discharge, during a transfer or readmission. The BCPDR also does not have a variable for maternal ICU admission, so we used procedures such as maternal intubation and ventilation as a proxy. The validity of this proxy in our database is unknown, but we speculate that it is likely specific but not sensitive, leading to incomplete capture of ICU admissions. Ongoing work to access vital statistics and hospital discharge data on ICU admission should improve the accuracy of these components. Nevertheless, our sensitivity analyses in which we quantified the impact of under-capture by tripling the rates of each of these events had only a modest impact on the overall measures of AOI, WAOS, and SI. We were unable to adjust the risks of WAOS and SI for patient case-mix in our inter-institutional comparisons. However, our finding that risk-adjusted AOI rates were reasonably similar to the crude AOI rates support the use of crude WAOS and SI. Finally, the AOI, WAOS, and SI were designed to monitor the safety and health care quality of labour and delivery, and are do not intended capture other types of pregnancy complications such as severe maternal morbidity or congenital anomalies. These indices should, therefore, be supplemented with other indicators if a more global assessment of the health status of pregnant women is desired.