Background

It is important for a health system to have an accsurate picture of overall (crude) and cause-specific hospital mortality. Although using mortality or risk adjusted mortality as an indicator of quality of care is contested because it is hard to adjust for case-mix or the severity of illness on arrival (case-severity) [1–4] the presence of 3.variation in mortality may point to possible inequalities in population health, access or resource provision that can be addressed. Yet, little attention has been paid to understanding and exploring hospital mortality and its variability in African settings, perhaps because routinely reported data are often of poor quality [5].

In this report our aim is to contribute to efforts to understand health system performance and describe mortality and its variability. We also explore whether simple clinical characteristics used to classify severity of common childhood illness are consistently associated with inpatient mortality. Demonstrating the latter provides a rationale for reinforcing their widespread clinical use and for considering them as components of a common data framework for paediatric admissions. A common data framework (potentially included in emerging electronic record systems) could improve our ability to characterise hospitals by their case-mix and case severity and inform health system strengthening efforts in support of universal access to quality health care.

Methods

Study setting

In 2014, Kenya had a gross domestic product of 1246 US dollars per capita after rebasing [6] and the under-five mortality was 58.3 per 1000 live births according to 2013 estimates [7, 8]. It has good immunization coverage of 90% for the 3rd dose of the pentavalent vaccine (introduced in 2002 and containing Diphtheria, Pertussis, Tetanus, Hepatitis B and Haemophilus influenzae type B antigens) and 85% for the 10-valent pneumococcal conjugate vaccine (PCV 10, introduced in 2011) [7].

The study takes advantage of a recently established clinical information network (CIN) comprised of 13 county referral hospitals. However, one county hospital was excluded from the analyses presented because of persistent problems with data collection. In total 12 facilities (11 counties) are therefore included in this analysis. The selection and geographic location of hospitals is presented in detail in panel 1, Table 1 and Fig. 1. In brief, CIN is a partnership between researchers, the Ministry of Health and paediatricians and is a pragmatic research database collecting patient level data from all paediatric admissions with aims at improving use of information in policy and practice.

Table 1 Characteristics of hospitals in the clinical information network
Full size table
Fig. 1
figure 1

Geographic location of hospitals. Black dots represent hospitals in the clinical information network and included in the analysis while the black lines represent county boundaries. Hospitals are clustered in the central and western regions consistent with where the majority of the Kenyan population lives

Full size image

Panel 1: Selection of the Clinical Information Network sites and Case Sampling

Kenya devolved health care provision to 47 county administrations in 2013 with the national Ministry of Health retaining responsibility for monitoring and evaluation amongst other areas. The clinical information network (CIN) was designed as a partnership between researchers, the Ministry of Health and paediatricians and is a pragmatic research database collecting patient level data from all paediatric admissions with aims at improving use of information in policy and practice. Twelve counties were first identified purposefully with the Ministry of Health to ensure the feasibility of the project while representing two main groupings based on the prevalence of malaria as an admission clinical diagnosis: high (>50%) and low (<20%). Within counties tertiary level facilities were excluded (found in 1 of the 12 counties) but public hospitals providing first referral level care (also called secondary level) within these counties and estimated to have at least 1000 paediatric admissions per year were considered eligible. One hospital was then purposefully selected from each county except in the largest urban county serving a population of over 3 million from which two hospitals were included. This resulted in two geographic clusters of hospitals (Fig. 1).

These hospitals were invited to join the proposed clinical information network (CIN) after its nature and purpose was explained to each hospitals’ management team and paediatric departments and their agreement sought. Characteristics of hospitals in the CIN are presented in Table 1.

In brief, the CIN collects core data that should be part of the routine health information system (RHIS) (the RHIS dataset) on all children admitted to hospital after their discharge (see below). In 10 of the 12 hospitals a comprehensive set of additional data were also collected on all admissions but due to high workloads in two hospitals (A and C) such comprehensive data were collected only on a random sample of 35% and 65% of the admissions respectively.

Study population

The population of interest for this report is restricted to children aged 2–59 months, the subjects of available, evidence-based clinical guidelines [9]. Hospitals joined the network in a staggered fashion from September 2013 over a period of 6 months, the period to March 2015 therefore provides at least one full calendar year for analysis from each hospital. In these hospitals, diagnoses at the time of discharge or death are clinician defined and informed by access to only very basic diagnostics (for example malaria blood slide but not blood culture) and post-mortems are done extremely rarely.

Data collection

Hospitals were encouraged to fully implement a structured pediatric admission record (PAR) that has been shown to improve documentation of core clinical characteristics at admission [10] and which was adopted by the Ministry of Health in 2010 as the admission encounter form for district hospitals [11]. Data were collected from the routine case record in an approach described in full elsewhere [12]. In brief, data were abstracted on the day following discharge from each child’s case record and entered directly onto a computer in a REDCap [13] database with in-built range and validity checks. Data clerks were trained centrally as a team prior to data collection in hospitals according to detailed, written standard operating procedures provided as a manual. At the end of every day before data were synchronized to a central database, the clerk checked on-site for errors, completeness and consistency with locally executed ‘cleaning’ programmes. Any inconsistencies or errors identified were corrected after verification from the case record. Throughout the study period clerks were coordinated and supervised by a research team member who telephoned approximately weekly and made visits approximately two-monthly when ongoing training to improve data collection was conducted.

Analysis

The Routine Health Information System (RHIS) dataset includes patient age, sex, weight, diagnoses and outcome. Secondary variables described below were generated from these. Age was categorised into 2–6, 7–11, and 12–59 months groups based on differences in observed risks of death in the dataset. The number of diagnoses made at admission (comorbidities) was categorised into no comorbidity, one comorbidity, two comorbidities and three or more comorbidities. Weight-for-age z-score (WAZ) measurement was classified according to standard WHO reference tables for age, weight and gender as follows: children with a Z score, > − 1 were classified as normal, children with a Z score < = − 1 and > −2 were classified as mildly malnourished, children with a Z score, <= −2 and > −3 as moderately malnourished, and children with a Z score < = − 3 as severely malnourished. The RHIS dataset was used to explore those associations between patient characteristics and mortality made possible if the routine health information system were functioning well. A secondary variable was generated to categorise hospital groups by malaria prevalence as an admission diagnosis (high >50%; low <20%). This grouping has some association with HIV prevalence and poverty levels (Table 1).

The comprehensive dataset included an additional checklist of clinical symptoms and signs recorded by the duty clinician on the paediatric admission record. These include danger signs and other signs promoted by Integrated Management of Childhood Illness guidelines [14] and those previously associated with all-cause or disease-specific mortality [15–19] that are used in national [20] and World Health Organisation [21] evidence-based guidelines to guide diagnosis, severity classification and treatment for the commonest causes of paediatric hospital admission (malaria, pneumonia, diarrhoea/dehydration, malnutrition, meningitis, anaemia and asthma). They are listed in Table 2. As there were few observations in each of the V (2%), P (4%) and U (<1%) categories of the AVPU scale these observations were re-categorised into Alert and Not alert.

Table 2 Available data by risk factor and hospital
Full size table

Statistical analysis

All patients without outcome data or with an implausible or missing date of admission, discharge or death were dropped from the analysis (see Fig. 2). Using the RHIS dataset we present hospital specific crude mortality rates and accompanying confidence intervals. To explore whether hospital mortality was associated with hospital identity, we used the RHIS dataset and fitted a fixed effects model with hospital but no other covariates and compared this to a null model using a likelihood ratio test (LRT). Hospitals were retained as fixed effects in all multivariable models because of significant associations with mortality and because we had only a small, non-random sample of hospitals (considering hospitals identities as random effects made no appreciable difference to results, data not shown).

Fig. 2
figure 2

Availability of data across the different models. Illustrates the number of observations used in each of the models listed

Full size image

The data available on clinical signs, symptoms and diagnoses are presented as hospital specific proportions in Table 2 and illustrate the variability across sites. We used logistic regression without adjustment to explore associations with mortality for each variable but without hospitals as fixed effects (Table 3 ). We then built multivariable models to explore associations of key clinical factors with mortality. We included malaria prevalence as a fixed effect in the multivariable models and explored for interaction between the risk factors and malaria prevalence. We used a multivariable model (model 1) based on the RHIS dataset to explore the ability of these data to explain mortality (Table 2). Model 2 included all variables in the comprehensive dataset (Table 2) in a complete case analysis including interaction terms for malaria prevalence. This included only 57% of all cases as a result of list-wise deletion of records with any missing data (see Fig. 2). The degree of missingness, represented as a proportion, varied by hospital and variable (Table 2). To address the missing data problem multiple imputation was used (panel 2) with the validity of results explored in sensitivity analyses (panel 3).

Table 3 Univariable and Multivariable analysis for associations with mortality
Full size table

Panel 2: Handling missing data

We explored and subsequently assumed a missing at random (MAR) mechanism as a basis for multiple imputation using the chained equation methods proposed by van Buuren [22] and Raghunathan [23]. Imputation was based on 100 iterations and 10 datasets as has been recommended for missing data rates of 10% -30% per variable[24, 25]. All variables in the RHIS and comprehensive datasets and identified interaction terms were included in the imputation procedures. To improve the power of the imputation model, we included auxiliary variables (history of cough and difficulty breathing) which may be clinically useful in diagnosis and are relatively well documented, and a variable denoting use of the paediatric admission record as this improves documentation and may therefore influence missingness [10]. We replicated analyses of associations with mortality including interaction terms after imputation in Model 3 (see Table 3).

In order to assess the plausibility of a MAR mechanism, we performed analysis under a Missing Not At Random (MNAR) assumption using pattern mixture models that included interaction terms. This proceeded as follows; first, we derived three missingness patterns amongst cases in the dataset: no missing data (57% cases); minimum 1 to maximum 3 variables per case with missing data (26% cases); >3 variables per case with missing data (17% cases). We performed multiple imputations and fitted the same multivariable models for each pattern independently. Thereafter, we pooled the estimates across the three patterns weighting by the proportions of individuals in each pattern per variable and compared these results with model 3 estimates estimated under a MAR assumption. We present the results of the MNAR analyses in Appendix.

Panel 3: Sensitivity analyses

We conducted various sensitivity analyses to explore the consistency of our results under different scenarios. First, we explored associations with mortality using the comprehensive imputed dataset but restricted the analysis to cases with only common childhood illnesses (malaria, pneumonia, diarrhoea/dehydration, malnutrition, meningitis, anaemia and asthma) to exclude possible effects of uncommon, high mortality conditions that might vary across place. Second, we conducted analyses restricted to the ‘best months’ of data collection by excluding the first 9 months (November 2013 to July 2014) for hospitals I and J and 7 months (February to August 2014) for hospital H to limit the scale of imputation. Third, we undertook analyses for data stratified according to whether cases were in high or low admission seasons. The estimates for associations between risk factors and mortality from these sensitivity analyses were clinically not appreciably different from those reported for Model 3 (data available on request).

Calibration and discrimination of the models was assessed using pseudo R-squared and area under the receiver operating curve (AUC) measures. Results from the univariable and multivariable analyses are reported as crude and adjusted odds ratios respectively with corresponding 95% confidence intervals (CIs) adjusted for clustering within hospitals. All analyses were undertaken using Stata v13 (StataCorp, Texas, USA).

Results

A total of 44,314 children were admitted into the CIN hospitals from September 2013 to March 2015, of these 33,741 (76%) were aged 2–59 months, had outcome data and a plausible date of admission or death. Characteristics of children by hospital and overall are presented in Table 2 . The overall cluster adjusted crude mortality across hospitals was 6.2% (95% CI 4.9 to 7.8; range 2.1% - 11.0%) with five-fold variation across hospitals while the risk-adjusted mortality rate derived from Model 3 was 6.2% (95% CI 4.7 to 7.6; range 3.0% - 9.4%) (Fig. 3a). Hospital identity was significantly associated with mortality (likelihood ration test, LRT <0.001 when compared to the null model). The distribution of risk factors per admission varied by hospital and outcome with 85% of children who survived having 3 or fewer risk factors while of those who died 53% had more than 3 risk factors (see Fig. 3b).

Fig. 3
figure 3

Crude and risk-adjusted mortality and patient risk profiles by hospital. The top panel Fig. 3a represent crude (light grey) and risk-adjusted mortality rate (dark grey) across hospitals with accompanying 95% confidence intervals. The bottom panel Fig. 3b represent the median and the 25th and 75th interquartile ranges for number of risk factors per patient stratified by mortality. Hospitals are ordered by malaria endemicity (low A, B, C, D, E, F, G; high H, I, J, K, L)

Full size image

The amount of missing data varied by variable and hospital (see Table 2 and panel 2). Pooled estimates of association weighted by proportion across missingness patterns were similar to those from model 3 (imputed data assuming MAR) and are presented in Appendix. These findings provide support for assuming that data were missing at random.

Clinical risk factors for mortality

All characteristics included in the RHIS dataset, derived covariables and all primary symptoms and signs included in the comprehensive data were significantly associated with mortality in univariable analyses except history of fever with anticipated (Table 3) direction of effect. Although we utilised alert vs not alert in risk adjustment exploratory univariable analysis illustrated an increasing risk of mortality with a decreasing conscious level; with alert as the reference category risks were V OR 4.61; 95% CI 3.54–6.00, P OR 9.31; 95% CI 7.84–11.05 and U OR 17.59; 95% 13.42–23.04.

Multivariable model 1, using RHIS data had an AUC of 0.73 and pseudo-R2 of 0.09 suggesting poor to modest model fit (assuming a cut off for good model fit for pseudo-R2 of 0.20 and above [26]) and that they are not likely to be suitable for understanding hospital populations’ risk of mortality. The complete case analysis model (model 2) and analyses after imputation (model 3), including malaria prevalence interactions, had better model fit (pseudo-R2 of 0.25; AUC of 0.86 and pseudo-R2 of 0.24; AUC of 0.85 respectively).

Estimates of association from complete case analysis and data sets using all admissions after imputation were similar in magnitude and direction of effect across all the clinical risk factors and we therefore present results from model 3 using imputed data as this makes maximum use of available data. These analyses (Table 3) show in all cases where interactions were not found that risk factors remained significantly associated with mortality except vomiting everything and number of comorbidities. Male gender (OR 0.67; 95% CI 0.57–0.79) and history of fever (OR 0.76; 95% CI 0.62–0.92) were associated with protection while reduced conscious level was strongly associated with mortality (OR 3.95; 95% CI 3.16–4.95). Tests for interactions between risk factors and malaria prevalence were significant for ability to drink, stiff neck, severe wasting, oedema, age group and weight-for-age z-score.

For covariables with significant interactions, estimates of association for having a stiff neck and severe wasting in low malaria prevalence hospitals had consistent but lower magnitude directions of effect than in high malaria prevalence areas (OR 1.53; 95% CI 0.81 to 2.90 vs 3.92; 95% CI 2.15 to 7.16 and OR 1.66; 95% CI 0.95 to 2.87 vs OR 2.54; 95% CI 1.70–3.82 respectively) and were not significant (Table 3) perhaps reflecting a loss of power (see Table 2). The estimates of association for being able to drink, oedema of malnutrition and age group remained significant in both malaria prevalence settings with consistently lower magnitudes of association in low malaria prevalence settings (OR, 0.65 vs 0.44, 3.13 vs 1.88, and 0.43 vs 0.33 (7–11 months) and 0.37 vs 0.29 (12–59 months) respectively). For weight-for-age z-score the estimate of effect was also attenuated in the low malaria prevalence hospitals (OR 2.09; 95% CI 1.22–3.56 vs OR 3.01; 95% CI 2.09–4.35). Signs associated with dehydration were consistently associated with mortality. For three hospitals (B, C and D) a persistent association with mortality was observed within these multivariable models (ORs 1.69, 1.90 and 1.99 respectively, Table 3).

Discussion

All-cause mortality is highly variable across only 12 hospitals even within a common age group. Variation in mortality was associated with the proportion of children with multiple risk factors, something that cannot be determined using data from existing routine health information systems. This variation in risk factors at presentation might be linked to the varying number of comorbidities at presentation resulting from overlapping syndromic diagnoses. This finding of major variation in the risk profile of children admitted to hospitals is important but rarely highlighted in existing research literature from low-income settings. The variation of mortality with hospital identity is probably explained by associated variability in underlying risk factors (such as malaria and HIV prevalence, socio-economic status, nutrition and access) that influence case-mix and case-severity. Although, these data exist at a population level, adjusting for these parameters would require a large number of hospitals in more defined, smaller regions. As such we have refrained from adjusting for these parameters in our models due to the risk of ecological fallacy where population estimates do not necessarily apply to the population seeking care. In addition, care seeking patterns in Kenya vary across populations and individuals with some of the patients accessing care directly from hospital while others are through referral from primary health care services but these data are rarely documented and hence difficult to disentangle within this population. Thus, although there is a standard policy guiding the provision of PHC in Kenya and how patients might progress through the system anecdotal evidence suggests procedures are rarely followed in practice and there are limited data from primary care on access and care seeking behaviors.

Mortality may also potentially be influenced by differing availability of resources across hospitals and variation in care practices shown to exist in other studies [27, 28]. More comprehensive datasets from a larger number of hospitals, as are being used in high income settings [29, 30], would however be required to try and determine the degree to which quality of care and local context are associated with inpatient mortality in low-income settings. Even then the use of mortality to infer anything about quality of care specifically is contested [1, 31]. What is more pertinent to low-income contexts is that such variation in mortality should prompt thinking on where system strengthening efforts may be most needed.

In the multivariable analyses, we demonstrate that majority of clinical risk factors included in best-practice guidelines were associated with mortality irrespective of diagnosis. This approach is important due to the inability to confirm diagnoses in these settings. For example, we have previously reported that basic investigations like blood culture, CSF analysis and urine culture are not routinely available [28] and there is no access to tests of inflammatory markers, for biochemical derangements or for detection of other pathogens except for malaria and HIV. Thus, an approach that is agnostic of diagnosis may be the more useful approach at scale. Prior demonstrations of association are typically from single sites or focus on specific diseases [15, 16, 19, 32]. We believe this is the first report of the value of these clinical risk factors from multiple settings, across all cause admissions, with routine observations made by large numbers of clinicians. As malaria prevalence and other attributes of the hospitals’ setting may be correlated, including for example proportion living in poverty and hospital size, we cannot attribute the influence of malaria prevalence on the behaviour of risk factors to malaria as a disease. However, findings suggest that future efforts to explore variability in mortality or develop risk prediction approaches should take the prevalence of malaria into account.

Our data support the practical, day to day use of these clinical factors in identifying children who may be prioritised for attention, specific treatments and review. Integrating such clinical factors into a low-income setting, patient-level prognostic score might be possible [16] replicating approaches in high-income settings [33]. However, the implementation of such scores in routine settings with limited human resources, high staff turnover and without computer assisted decision support would likely be very challenging as even basic job aides are somewhat slowly adopted [28]. These clinical features might however be used to characterise risk profiles of hospital populations aged 2 to 59 months. This might enable improved understanding of changes in mortality over time within sites using methods such as cumulative sum control charts (CUSUM) [34–37] where risk-adjustment may facilitate exploration of variation in a single institution’s performance [1, 3, 31]. More pragmatically such risk profiles may help point to different health system challenges. High prevalence of cases with high risk factor density may suggest problems of access, late detection or delayed referral. They may also point to particularly vulnerable populations within catchment areas, where malnutrition and poverty are prevalent for example, or demonstrate the impact of varying disease ecology that should be taken into account in resource allocation. Our findings suggest that existing routine health information systems data (such as DHIS2) [38] would be inadequate for this purpose, suggesting value in developing and implementing suitable common data frameworks.

The data we report need to be interpreted in the light of their limitations. First, is the missing data problem commonly associated with collecting data in routine settings. The Clinical Information Network worked with hospitals to promote data quality that improved over time but missingness varied across variables and hospitals. Thus, just less than 60% of cases were included in our complete case multivariable models. We used multiple imputation to allow use of all available data and undertook sensitivity analyses that suggest our findings are generally robust. An alternative approach would be using Expectation-Maximization (EM) algorithms to get maximum likelihood estimates [39]. Second, our sample of hospitals is small and non-random, with selection based on feasibility and efforts to represent diverse but not all epidemiological and socio-cultural contexts. One hospital that failed to provide reasonable quality data was excluded from these analyses. As such, due to the limited number of hospitals, our models did not sufficient power to explore potentially important factors at the hospital level. Thirdly, diagnoses are clinical and rarely informed by diagnostics while risk factors such as hypoglycaemia, hypoxemia or an individual’s HIV status could not be examined as these are rarely routinely evaluated. Fourth, we did not include more robust nutritional indicators like mid-upper arm circumference or weight-for-height z scores or account for vaccination status which may influence mortality because such data are largely missing. However, there is evidence that vaccination coverage is high in Kenya and we were able to use weight-for-age z scores for nutritional assessment.

Conclusion

In summary, all-cause crude and cluster adjusted mortality rate was highly variable across hospitals. Such variation is largely explained by variation in severity of illness at the time of clinical presentation, findings that point to underlying differences in population health and health system performance that will need to be explored. Our data supports the use of clinical risk factors drawn from guidelines in day-to-day use in prioritizing care and identifying children at the highest risk of death but also to develop risk adjusted mortality estimates across hospitals. We also demonstrate how having a large patient level dataset from multiple geographically diverse sites may improve our understanding of health system challenges and performance. Such work provides a learning platform for the design of common data frameworks that are relevant to clinical practice and might be incorporated into future electronic medical records (EMRs) that go beyond a focus on cost-accounting needs [40]. To maximise the future value of EMRs there is a clear imperative for researchers, clinicians, policy makers and health care managers to engage in their design so they enable health system performance monitoring at scale as is occurring in specific fields such as HIV care [41, 42].