Background

Quality improvement (QI) approaches are increasingly being used in low- and middle-income countries in efforts to improve service delivery and health outcomes. Much of the literature on QI approaches in these settings focuses on documentation of implementation and process evaluation [1,2]. A few studies have described scale-up processes for QI interventions in Ecuador [3,4], India [5] and South Africa [6,7]. All of these studies provide valuable information to guide countries and projects. However, documenting the impact of such approaches, both during pilot stages and at scale, is also important. This paper presents an impact evaluation of the scale-up phase of ‘Project Fives Alive!’, a national QI intervention in Ghana that aimed to improve maternal and child health outcomes. The project was implemented by the National Catholic Health Service and Institute for Healthcare Improvement in collaboration with the Ghana Health Service. The project design has been described previously in detail [8], and a prior evaluation study documented the impact of the pilot phase of the project [9].

The objective of ‘Project Fives Alive!’ was to assist and accelerate Ghana’s efforts to achieve Millennium Development Goals (MDGs) 4 (reducing under-five mortality) and 5 (reducing maternal mortality). Though Ghana did not meet the targets for MDG-4 and MDG-5, large improvements were made. In 2013, Ghana reported having a maternal mortality ratio of 380 maternal deaths per 100,000 live births, a large decline from the estimate of 760 maternal deaths per 100,000 live births in 1990 [10]. In 2015, the under-five mortality rate was estimated at 62 under-five deaths per 1000 live births compared to 122/1000 in 1990 [11].

‘Project Fives Alive!’ began in July 2008 with an innovation and testing phase, Wave 1, which included 27 health facilities in Northern Ghana. These facilities were purposively selected to reflect a mix of government facilities and faith-based facilities, which are affiliated with a religious institution. Wave 1 provided an opportunity for the implementation team to develop a package of locally identified and tested change ideas (interventions) focused on improving care seeking and care giving for mothers and children. Following Wave 1, the project rapidly introduced the locally developed interventions (changes) through a subsequent scale-up phase, Wave 2, to all government and faith-based facilities in Northern Ghana. Wave 2 included over 800 health facilities and covered the time period of September 2009 to March 2013.

‘Project Fives Alive!’ used the Model for Improvement and its underlying QI approach of identifying gaps in performance and the process failures that led to those gaps. The next step is identifying and testing simple low cost change ideas (or interventions) that can be employed to address those failures [12]. The process of generating, testing and sharing those ideas is accelerated through the Institute for Healthcare Improvement’s Collaborative Breakthrough Series Model that brought multiple sub-district or facility QI teams together repeatedly to share knowledge for improving performance. These teams formed Improvement Collaborative Networks at the district level [13] within specific geographic locations. During Wave 1, QI teams were formed at the level of the facility, while in Wave 2 teams were formed at the sub-district level such that all health centres and health posts within a sub-district contributed team members to form one team. Due to the higher volume and higher acuity of patients in hospitals settings, each hospital in Wave 2 formed its own team. Both Waves used the same basic Breakthrough Series approach; the QI teams attended four Learning Sessions (structured workshops led by project staff) where they learned QI methods and had a chance to share progress and ideas with other QI teams. During the 4–6 months between the Learning Sessions, both Waves 1 and 2 included Activity Periods when QI teams conducted Plan-Do-Study-Act cycles, the primary mechanism for testing and implementing changes. These cycles involved small tests of changes followed by rapid evaluations and adaptions.

Though the basic approach was the same, there were some key differences in implementation between Waves 1 and 2. Due to the small scale of Wave 1 (n = 27) and focus on innovation and development of change packages, the project team spent considerable time coaching each QI team during the Activity Periods. However, due to the large numbers of facilities in Wave 2 and only a small increase in project staff, it was not feasible to continue with this programmatically intense strategy. Thus, district health staff undertook intensive training and performed more of the coaching activities under the close supervision of project staff.

The main objective of this paper is to determine whether ‘Project Fives Alive!’ influenced maternal and child health outcomes at scale. A secondary objective is to present a methodology of using facility-based routine health data for a large-scale impact evaluation.

Methods

We employed a quasi-experimental design with a multivariable interrupted time series analysis controlling for potential confounding factors, to understand the impact of the intervention. Outcome data for this analysis are derived from data measured and reported by the facilities, while independent variables come from facility and program records. A specific programmatic decision was made to use routinely reported data rather than institute a parallel project data collection system, since the intervention was designed to be sustainable and scalable. To support this decision, a major effort was undertaken to improve the timeliness, completeness and accuracy of the data being submitted to and reported by the Ghana District Health Information Management System (DHIMS), a system whereby facilities complete monthly reports of key indicators and these are compiled at the district and national levels.

A total of 744 facilities were included in our analysis. Some newer facilities could not be included because of a lack of pre-intervention data. Since the intervention has the potential to differentially impact health outcomes by facility type and also due to different degrees of missing data, the analyses are stratified by facility type: health posts (or first level facility), health centres, and hospitals. Ethics review approval was obtained by the Ghana Health Service and the University of North Carolina at Chapel Hill.

Outcome Data

This evaluation used data from January 2009 to March 2013. In the initial phase of Wave 2, facilities used paper-based forms to report on key outcome indicators in a system called DHIMS 1. These forms were compiled and entered at the district level and then electronically sent onwards to the national level. In January 2012, Ghana shifted to a complete electronic system, DHIMS 2, whereby facilities entered the data and submitted the forms directly to the national level.

The four outcome variables in this assessment were chosen based on relevance to the project, and three were also included in the Wave 1 impact evaluation. Each outcome variable studied and the exact metric we used to define the variable are described in Table 1. The maternal health variables are early antenatal care (ANC) and skilled delivery coverage. We were able to study coverage for skilled delivery because health facilities record births that occur both at home and in facilities. The child health outcomes included the percent of child welfare clinic (CWC) attendees who are underweight and facility-level under-five mortality (for hospitals only). Underweight is defined as low weight for age in comparison to WHO reference standards. Our definition encompasses both moderate (less than two standard deviations below the median of the reference) and severe underweight (less than three standard deviations below the median of the reference standard) [14]. We could not study neonatal and infant mortality because of data quality concerns stemming from changes in reporting from DHIMS 1 to DHIMS 2. Fewer facilities reported these mortality outcomes in the DHIMS 2.

Table 1 Outcome variables and their definition/metric
Full size table

Key independent variables

One of the two key independent variables was the interventions or change ideas implemented at a facility. For health centres and health posts, the change ideas were grouped into five categories – early pregnancy identification, four or more ANC visits, skilled delivery/immediate postnatal care (PNC), PNC on day 1 or 2, and PNC on day 6 or 7. Examples of these change interventions included community stakeholder meetings and registration of pregnant women by community volunteers for the early ANC; ANC defaulter tracing and visit time reduction for the four or more ANC visits; use of partographs and immediate checks of mother and newborn for the skilled delivery/immediate PNC; and home visits for both PNC interventions.

Hospitals had a separate set of change categories more suited to their patient loads and the presence of higher level staff. Hospital change categories were health education, targeting/engaging primary providers, training, triage and task shifting/nurse empowerment. The hospital changes were more broadly targeted than the health centre and health post changes since hospitals cover all types of services. Hospital changes were expected to improve maternal and child health by shortening visit times, prioritizing sick mothers and children, and improving communication between providers and pregnant women and mothers.

The other key independent variable was the implementation phase in which these improvement activities occurred – Wave 2a or Wave 2b. The earlier phase, Wave 2a, included the majority of facilities, whereas Wave 2b included the later set of facilities to engage in implementation. We study this variable to understand whether all facilities benefit equally or whether facilities that initiate implementation earlier, benefit more.

Control variables

The facility-level control variables included in this analysis were the type of health facility (hospital, health centre or health post) and affiliation of the health facility (government or faith based). A dummy variable was also included to represent the project officer assigned to work with a particular QI team. We also included as control variables profession of the QI team leader and number of QI team members. Since health insurance, particularly Ghana’s National Health Insurance Scheme, may be a potential confounding factor, a monthly time varying health insurance control variable (which was measured as the percent of outpatients who had insurance) was included.

Analysis

Descriptive analysis

In our descriptive analysis, we present a comparison of the pre-intervention, transition phase and post-intervention means of the outcome variables. The pre-intervention phase was defined by the project implementation team as the period of time before Learning Session 1, when QI teams were still learning the methodology of testing changes. The transition phase was the time period from Learning Session 1 to the end of Learning Session 2, which was considered a time when teams had just completed the training needed to fully implement the QI approach. The post-intervention or full saturation phase began at the very end of Activity Period 2 and was considered the cut-off point when the QI teams were expected to have the skills and knowledge to fully implement change ideas.

Missing outcome data

The unit of observation for the outcome data was facility-months. Each facility had several months of data. The number of facility months varied by outcome because not all facilities reported on all outcomes, and some facilities were new and were not in existence during the early time points or did not report on a particular outcome at exactly time 1. In addition, missing data, which was defined as an outcome not reported in a particular month once a facility has initiated reporting, was also responsible for some of the differences. Furthermore, with Ghana’s change in facility-level reporting of outcomes in January 2012, some facilities no longer provided denominators needed for the skilled delivery outcome. For these facilities, we ended their observation interval in December 2011 for skilled delivery to avoid considering the data points from January 2012 onwards as missing. Figure 1 presents the amount of missing data both by facility and outcome. Hospitals and health centres had relatively low amounts of missing data for the maternal health outcomes, whereas health posts had slightly more missing for these outcomes. Missing data for attendance of underweight infants at CWCs varied from 31% for health centres to 41% for hospitals. Hospitals had 43% missing data for under-five mortality.

Fig. 1
figure 1

Percent of facilities with missing and non-missing data for health outcomes

Full size image

Multivariable time series analysis

To study the impact of ‘Project Fives Alive!’ on the outcomes, we employed a multivariable interrupted time series regression analysis. This type of analysis answers the question of whether an intervention is associated with a change in the underlying trend for the outcome of interest after controlling for key variables [15]. The methodology of using repeated or monthly observations from the same facilities both pre- and post-intervention offers a strong evaluation design [16]. Data came from the period of January 2009 to March 2013, and the first set of facilities did not reach full implementation until July 2010. It was thus possible to establish underlying trends using the pre-intervention and transition phase data. In this analysis each facility served as its own control with the pre-intervention trend compared to the post-intervention trend.

In our model, there are two key parameters that were of interest – the immediate impact of the change category and the longer term impact or change category trend. Adding the coefficients from these two parameters yields the overall effect of the change category. Our model also included a quadratic term to account for a potential non-linear trend. A detailed description of the regression model and equation are presented by Singh et al. [9]. For each outcome variable, several multivariable regression models were run with relevant change categories included in separate models. Not all change categories were expected to have an effect on all four outcomes. For example, the PNC changes would not be expected to have an effect on skilled delivery, and thus regressions with these change categories are not presented for skilled delivery. A separate set of regressions to study the effect of program intensity, defined as the monthly number of change categories tested, was also run. These models also controlled for the independent variables presented earlier.

Due to the amount of missing data and the presence of both serial autocorrelation and clustering, we used generalized estimating equations (GEE) to run the regression analyses. GEE uses all data that is available and assumes data is missing completely at random, which is a plausible assumption for these monthly facility-level data. Autocorrelation and clustering violate the ordinary least squares assumption of uncorrelated error terms, biasing the standard errors when using standard linear regression. GEE is an extension of the quasi-likelihood approach used in generalized linear models and is often applied to modelling longitudinal data [17–19].

In addition, a sensitivity analysis was conducted as a check against our results from the main GEE analysis. In these analyses, single imputation was used to either (1) impute all missing values or (2) impute missing values only for facilities with less than 25% of their observations missing. The imputation was conducted by taking the average value of the nearest non-missing preceding and succeeding values. Because the results of our sensitivity analyses generally corroborated results from the main model, only results for the main model are presented.

The number of observations for each regression model varies slightly by facility type due to the varying amounts of outcome data available. In addition, not all independent variables were available for each facility. Comparisons of mean outcomes for facilities that have all control variables and those that do not were made and found not to be significantly different.

Results

Descriptive presentation of the independent variables

All independent variables are presented in Tables 2 and 3. The majority of facilities were health centres (45.5%) and health posts (50.4%), while only 4% were hospitals. Ninety-two percent of the facilities were government affiliated and 8% were faith based. A total of seven project officers were part of the Wave 2 program team, and the average percent of insured patients at a facility was 78%. The PNC change intervention activities were the most common changes implemented in health centres and health posts, while triage was the most common change category in hospitals. Eighty-seven percent of facilities were part of Wave 2a, and 13% were part of Wave 2b.

Table 2 Descriptive statistics for control variables and independent variable phase
Full size table
Table 3 Descriptive statistics on change category variables, by facility type, n (%)
Full size table

Descriptive analysis

The comparison of means for the pre-intervention, transition and post-intervention phases is presented in Table 4. Overall, there are improvements in the maternal health outcomes over time. At the aggregate level, there is an increase in early ANC from 37% to 42% to 48% from the pre-intervention phase to the transition phase to the post-intervention phase, respectively. Overall, skilled delivery is at 42% in the pre-intervention phase, increases to 47% in the transition phase and then further increases to 51% in the post-intervention phase. In terms of the child health outcomes, there was an overall increase in the percent of underweight infants attending CWCs, from 2% in the pre-intervention phase to 8% in the post-intervention phase. Under-five mortality decreases from 36/1000 in the pre-intervention phase to 29/1000 in the post-intervention phase.

Table 4 Means of the outcome variables by implementation period and facility type
Full size table

Time series analysis

Health posts

Three of the categories of change interventions – early ANC (β = 0.3540, P < 0.01), four or more ANC visits (β = 0.2882, P < 0.05) and skilled delivery/immediate PNC (β = 0.2822, P < 0.01) – were significantly and positively associated with the skilled delivery outcome (Table 5). Facilities that tested these changes saw 28–35% higher rates of skilled delivery than facilities that did not test such changes. The corresponding trend variables had small positive associations with the skilled delivery outcome, suggesting that the initial positive effect continued over time, but this trend effect was significant only for the early ANC and skilled delivery/immediate PNC changes.

Table 5 Results of generalized estimating equation regressions of health outcomes on change categories at health posts
Full size table

In terms of findings for child health, both PNC change categories were significantly associated with a greater percentage of underweight infants among CWC attendees. The β2 coefficient was 0.0667 at P < 0.01 for the PNC day 1 or 2 change category, and the β2 coefficient was 0.0578 at P < 0.01 for the PNC day 6 or 7 change category. The corresponding trend variables were not significant.

There was one significant finding for the measure of intensity in the health post analysis, namely the monthly number of change categories tested (Table 6). Intensity was significantly associated with a lower percentage of underweight infants among all CWC attendees (β = –0.0092; P < 0.001).

Table 6 Results of generalized estimating equation regressions of health outcomes on intensity for health posts and health centres
Full size table

Health centres

None of the specific change categories were associated with the early ANC and skilled delivery outcomes (Table 7). Both PNC change categories, however, were associated with an increased percent of underweight infants among attendees at CWCs (β = 0.0701, P < 0.001 and β = 0.0452, P < 0.001, respectively). The trend variables were significant and positive, indicating that the initial increased effect was maintained. Wave 2b was significantly and negatively associated with attendance of underweight infants at CWCs in the models with the PNC change categories.

Table 7 Results of generalized estimating equation regressions of health outcomes on change categories at health centres
Full size table

There were two significant associations between the measure of intensity and the outcome variables (Table 6). A greater number of change categories tested was significantly associated with increased skilled delivery (β = 0.0089, P < 0.05) and a smaller percentage of underweight infants among child wellness attendees (β = –0.0088, P < 0.001). There was a significant association in the intensity models between Wave 2b and decreased underweight infants at CWCs.

Hospitals

There were several significant associations between the change categories in the regressions for hospitals (Tables 8 and 9). Facilities that tested a health education change had 58% higher rates of skilled delivery compared to facilities not testing this change category (β = 0.5753, P < 0.05). The trend variable was slightly positive and significant, indicating that the increase in skilled delivery was continued although at a lower level than the initial increase. The triage change category was associated with a 42% increase in early ANC (β = 0.4236, P < 0.05) and a 50% increase in skilled delivery (β = 0.4989, P < 0.05), and the trend variable for the latter indicated a slight but significant increase over time (β = 0.0004, P < 0.05). Across all outcomes, there were no significant associations with the implementation phase variable in the hospital settings.

Table 8 Results of generalized estimating equation regressions of maternal health outcomes on change categories at hospitals
Full size table
Table 9 Results of generalized estimating equation regressions of child health outcomes on change categories at hospitals
Full size table

Greater intensity was significantly associated with two of the outcomes in hospitals (Table 10). Intensity was associated with a 0.9% decrease in underweight infants attending CWCs (β = –0.0093, P < 0.01) and a 0.4% decrease in under five-mortality (β = –0.0038, P < 0.05). Once again there were no significant associations for the implementation phase variable.

Table 10 Results of generalized estimating equation regression of health outcomes on intensity for hospitals
Full size table

Discussion

As more low- and middle-income countries implement QI projects to improve health outcomes, there is a need to evaluate the approaches both during pilot and scale-up phases. Evaluations of pilot phases can help demonstrate the evidence needed to justify scale-up and/or can provide valuable information to inform implementation modifications for the scale-up phase [20]. Due to the magnitude of scale-up phases and the difficulty of finding control or comparison groups, innovative evaluation approaches are needed that can take advantage of existing monitoring data [21,22]. In this paper, we present an innovative evaluation of the scale-up phase of ‘Project Fives Alive!’ using data from Ghana’s routine health information system supplemented by facility characteristics and program records.

Findings from the evaluation indicated a positive effect of ‘Project Fives Alive!’ on key maternal and child health outcomes. There was evidence of some sustained program effect on underweight infants attending CWCs and skilled delivery as was seen in Wave 1 [9]. All the maternal health-focused change categories were associated with an increase in skilled delivery for health posts, and the health education and triage change categories were associated with the early ANC and skilled delivery outcomes for hospitals. Greater intensity was associated with increased skilled delivery for health centres.

There were positive effects of the PNC change categories in getting more underweight infants into care in health posts and health centres; however, greater intensity was also negatively associated with the percentage of underweight infants at CWCs across the facility types. These differing findings need a nuanced explanation. It could be that the PNC change categories initially increased care-seeking of caregivers of underweight infants. Over time, as these facilities implemented more changes and more fully incorporated the QI approach into their daily work, there could have been overall improvements in the health and nutrition of children in the catchment area, leading to a lower percentage of children in facilities who were underweight. These findings on underweight children are important given that under-nutrition is estimated to be an underlying factor in 45% of under-five deaths [23].

In Wave 1, there were no significant associations between the change categories or intensity with mortality. Perhaps due to the longer time period of Wave 2 compared to Wave 1 (51 months versus 21 months) we see evidence of impact on mortality for Wave 2. In hospitals, greater intensity was associated with slightly decreased under-five mortality. As health providers engaged more fully in the QI approach over time, they may have been able to improve the quality of services provided and/or increase early care-seeking such that mortality declined.

There were few significant differences by phase of implementation, indicating that all facilities benefited from the intervention. In ‘Project Fives Alive!’ the first set of Wave 2a reached full saturation in March 2010 compared to March 2011 for the first set of Wave 2b facilities. This is an important finding given that scale-up strategies of many projects need to have a phased approach to attain broad reach.

There are several limitations to this analysis, including our inability to study population-level mortality. We only had data on facility deaths and not deaths that occurred in communities. In Ghana, as in many low- and middle-income countries, many under-five deaths occur at home or in non-facility environments. An additional data challenge is that we could only study skilled delivery until December 2011 for a large number of facilities because of the change in reporting. Finally, it is difficult to find comparison groups for the evaluation of a scale-up phase of a project, and our analysis lacked such groups. We were able to use each facility as its own control in an interrupted time series analysis with additional control for potential confounding factors, including program and facility characteristics. We also controlled for National Health Insurance Scheme registration, which also has a strong focus on maternal and child health. The use of repeated monthly outcome data from each facility, both pre- and post-intervention, offers a strong evaluation design [16]. We cannot, however, completely rule out the possibility that other ongoing maternal and child health initiatives could have also influenced the results.

Conclusion

Findings from the scale-up phase of ‘Project Fives Alive!’ indicate program effects on the key maternal and child health outcomes studied, including reduced under-five mortality. The QI approach of identifying barriers to care and care-seeking with local, simple and inexpensive solutions has demonstrated impact at scale and should be considered a feasible approach for improving maternal and child health outcomes in other low- and middle-income settings. We also demonstrate the feasibility of using existing outcome data in a multivariable time series analysis to evaluate the scale-up phase of an intervention.

Abbreviations

ANC, Antenatal care; DHIMS, District Health Information Management System; GEE, Generalized estimating equations; MDG, Millennium Development Goal; PNC, Postnatal care; QI, Quality improvement