Background

The Evaluation of Processes of care and the Outcomes of Children in Hospital (EPOCH) study is a cluster-randomized trial comparing the effect of the Bedside Paediatric Early Warning System (BedsidePEWS) with standard care. The outcomes are clinical, processes of care and resource utilization. Here we describe the rationale for the trial, the intervention and the outcomes used, conflict of interest management and the potential impact of the study.

The problem and current treatments

Cardiopulmonary arrest occurs in 0.1-20/1000 children in hospital wards [1-4], with hospital survival of 27-50% [5-7]. Despite optimal resuscitation and post-arrest care, in-hospital cardiac arrest is associated with significant mortality and acquired morbidity in survivors [3,6,8-11], making prevention the best strategy [6,8,12-23].

Prevention of near and actual cardiopulmonary arrest

Prevention of near and actual cardiopulmonary arrest in hospitalized children is a patient safety imperative. Prevention is contingent upon the timely identification, referral and treatment of children whose conditions are clinically deteriorating [24,25]. Preventative strategies have focused on the latter portion of this sequence, primarily through improved access to ICU expertise through deployment of Medical Emergency Teams-Rapid Response Teams (MET-RRT). However, studies addressing methods to identify those in need have been few and of limited quality [26].

Emergency response teams

To date, MET-RRT have not fulfilled their promise. In adult inpatients, following several positive [27-29] and negative [30-33] single-center studies, a 23-hospital, cluster-randomized clinical trial compared implementation of un-validated calling criteria plus MET-RRT with standard care (no MET-RRT, no explicit calling criteria). The primary outcome was a composite of unexpected death, unplanned ICU admission and cardiac arrest. This negative trial [34] provides several important lessons: (1) calling criteria were met in <50% of patients with urgent ICU admission, suggesting that the criteria did not identify patients at risk. (2) The number of patients without unexpected death, unplanned ICU admission or cardiac arrest, who met one or more of these calling criteria is unknown (similar to pediatric studies) [35-39]. (3) Approximately 90% of the patients who met criteria were referred to ICU teams in both MET and control hospitals, suggesting that the calling criteria added little to the standard model.

In children, only before-and-after studies have been published [35-39], of which one is multi-center [40]. The two studies showing reduced all-cause hospital mortality were performed over 6 [36] and 7½ years [38], influenced the findings of a systematic review of pediatric MET-RRT [41], which include no RCTs, and reinforced the potential for description of time-related improvement [42,43] to be ascribed to MET-RRT. Furthermore, the additional benefit from MET-RRT without appropriate ‘afferent limb’ identification mechanisms may be questionable in hospitals where the efferent limb response-ICU expertise is readily available [25].

Mechanisms to identify children at risk for near and actual cardiopulmonary arrest

There are eight published pediatric mechanisms to identify children at risk for cardiac arrest [1,35,37,44-47]. The development of most has been methodologically limited [48]. There are four published calling criteria [35,37,46,47]. The Cincinnati criteria include subjective elements and have been modified [35,49]. The Baltimore criteria are subjective and/or triggered by acute medical diagnoses [37]. The Bristol tool is subjective and uses thresholds for the airway (nebulized epinephrine, ‘tiring’) and disability [46]. The Melbourne criteria identified 10/24 (42%) patients who had cardiac arrests in their hospital [38] and similar proportions in other hospitals [40,50].

There are four published pediatric scores [1,44,45,48]. The Brighton score uses behavior, circulatory (color, relative tachycardia) and respiratory (relative tachypnea) domains and ‘persistent vomiting following surgery’ with a range of 0 –11 [44]. Evaluation of this score in a cohort of 2979 children in Cincinnati Children’s Hospital found a sensitivity of 78% and specificity of 82% [49]. In the development cohort, the Cardiff and Vale score had a sensitivity of 69% and specificity of 90%. Inspection of the score reveals that children in the first months of life may be expected to routinely score at or above the threshold (2 points for systolic blood pressure <70 mmHg and respiratory rate >50 breaths/min) [45].

The BedsidePEWS

The BedsidePEWS score differentiated between children who were urgently admitted to PICU and ‘well’ hospitalized children, with at least 1 h notice with a sensitivity of 83% and specificity of 95% [48]. In a multi-center validation study of 2074 patients the score performed well in sub-populations [51]. Among ‘case patients’ the score increased over time leading up to an urgent PICU admission or code-blue event, and was independent of the number of risk factors for cardiac arrest. [51].

The score was reliably calculated with an intraclass correlation coefficient of 0.92 and was associated with statistically significant improvements in clinical and process of care outcomes [52]. The quality of development, validation and evidence of reliability supports the conclusion that the BedsidePEWS score is equivalent or superior to others and provides a rationale for its use as the intervention in EPOCH (Table 1).

Table 1 Differences between BedsidePEWS and other severity of illness scores
Full size table

The objectives of EPOCH are to evaluate the impact of the BedsidePEWS on early identification of children at risk for near and actual cardiopulmonary arrest, hospital mortality, processes of care and ICU resource utilization. We hypothesize that the BedsidePEWS improves early detection of critical illness, reduces mortality, improves processes of care and does not increase healthcare resource utilization.

Methods

EPOCH is a cluster-randomized trial comparing the BedsidePEWS with standard care (no severity of illness score) in hospitals with a pediatric intensive care unit (ICU). The unit of randomization is the participating hospital. Eligible hospitals provide care for more than a total of 200 eligible inpatient admissions in eligible inpatient wards each year, have specialized pediatric physicians and have at least one PICU. Eligible inpatient wards are defined as areas where care is provided to children who are admitted to the hospital, other than the NICU, PICU, operating rooms and other designated areas where anesthetist-supervised procedures are performed. Emergency departments that care for admitted patients may be regarded as an eligible ‘ward’ (Table 2). Eligible patients are >37 weeks gestational age and <18 years, and they have received care in an eligible inpatient ward. To ensure that major system changes do not introduce bias, we will exclude hospitals that plan to introduce or discontinue an MET-RRT during the study, are already using a severity of illness score in wards or consider randomization unacceptable. Hospitals with a pre-existing MET-RRT are eligible to participate.

Table 2 Study definitions
Full size table

Intervention: BedsidePEWS

The BedsidePEWS documentation-based system of care replaces existing documentation systems for vital signs in inpatient ward areas in hospitals randomized to the intervention. There are four elements in the system: (1) the BedsidePEWS score; (2) the BedsidePEWS documentation record; (3) score-matched care recommendations, developed using the responses of 280 healthcare professionals (80 community, 200 referral) surveyed to determine ‘reasonable’ care in each of the domains of vital sign assessment, secondary review, continuous monitoring and ICU consultation, and customized for each implementation; (4) the education program, developed by two nurse educators to support implementation and maintain expertise, including the BedsidePEWS Instructor course and frontline education program.

Randomization

Participating hospitals are randomized in a 1:1 ratio to intervention or control. Computerized randomization is balanced by the use of two strata of hospital size: <200 vs. ≥200 eligible inpatient ward beds. Allocation is concealed until the start of the study measurements and is revealed in the 2nd week after the start of data collection at each site (Additional file 1).

Primary outcome: all-cause hospital mortality

The primary outcome is all-cause hospital mortality in eligible inpatients who were cared for in an eligible inpatient ward at some point during their hospital stay. This includes anticipated deaths in children with ‘do not resuscitate’ orders. Mortality events in children cared for exclusively in the PICU, NICU or emergency department, or combinations thereof, are excluded.

Secondary outcomes: clinical

The main secondary outcome is the significant clinical deterioration event (SCDE), a composite outcome comprised of the treatment(s) provided or death prior to transfer from an inpatient ward. An SCDE is defined as the provision of significant pre-specified respiratory or circulatory therapies or cardiopulmonary resuscitation in the 12 h before transfer from the inpatient ward or during the hour after transfer, or death without a DNR order in an inpatient ward (Table 2).

EPOCH includes eight additional clinical outcomes (Table 3): (1) the nature of SCDE rated on the seven-point Children’s Resuscitation Intensity Scale (Table 4), which measures the timing of interventions in children for whom active resuscitation is anticipated; (2) potentially preventable cardiac arrest in patients experiencing a cardiac arrest while in an eligible inpatient ward. This excludes patients with a preceding DNR order. Potentially preventable is defined as the degree to which ‘events may have been avoided given the application of reasonable current (2011-13) standards of practice by an average practitioner and system anticipated to manage the condition in question’ [53-57]. Events with a consensus rating from two expert reviewers of ≥4/6 are defined as potentially preventable cardiac arrests (Table 5) [55,56]. (3) Unplanned hospital re-admission; (4) urgent PICU re-admission, both within 48 h of discharge. Finally, in all patients admitted urgently to the PICU from a hospital inpatient ward measurement was made of: (5) the predicted risk of mortality (PIM2 score) [58,59], (6) PICU mortality, (7) organ dysfunction (PELOD score) [60] and (8) ventilator-free days to 28 days. Urgent PICU admission is defined as departure from the inpatient ward within 6 h of the decision to admit to the ICU (Table 2). This definition reflects the need to pragmatically distinguish between urgent and elective admission and the recognition that most urgent admission occurs within 1-2 h of the decision to transfer to an ICU.

Table 3 EPOCH secondary outcomes
Full size table
Table 4 Clinical deterioration events
Full size table
Table 5 Potential preventability criteria
Full size table

Secondary outcomes: process of care

The study includes five processes of care assessments: (1) ‘Stat’ calls. This was formally defined (Table 3), and a consistent approach to data collection was applied throughout the study, recognizing that cultural and other factors may contribute to considerable inter-hospital variability. (2) Code Blue calls for immediate medical assistance of the resuscitation team and equipment. (3) Urgent consultations (within 15 min) to the ICU or MET-RRT. (4) Using documentation from five randomly (central computerized randomization) selected patients each week, physician visits, the nurse:patient ratio and the use of continuous monitoring are described. (5) Frontline staff complete the documentation and interaction survey (2 pages, <10 min to complete) that describes their perceptions of the documentation system and the nature of interactions with physicians (Additional file 2: CRF). This custom-built survey was developed and used in our pilot evaluations of BedsidePEWS. It has not been formally validated.

Secondary outcomes: resource utilization and decision-maker perceptions

Resource utilization is assessed in all hospitals by measurement of hospital length of stay and for patients undergoing urgent ICU admission, ICU length of stay and the days of use of the ICU technologies: mechanical ventilation, hemodialysis, ECMO and nitric oxide. A purpose-built survey of the perceptions of hospital decision-makers will be conducted 3 months after the end of the 12-month intervention period (Table 3) to describe the experience of the decision-maker in participating in a hospital-wide cluster RCT and their expectations of its results.

All outcomes are assessed prospectively. The first 26 weeks are baseline data. Then following a 5-week run-in period in intervention hospitals or break from data collection in control hospitals, outcomes are assessed prospectively for 52 weeks (Additional file 1).

Sample size and assumptions

Power calculations are based on population estimates derived from 14 local hospitals’ data using a published method for cluster RCTs [61]. For all-cause mortality we found a baseline rate of 5.1/1000. The steering committee agreed that a mortality reduction of less than 1/1000 admissions would not be a compelling reason to modify practice. A study with 20 hospitals can show an 18% relative risk reduction in mortality (absolute risk reduction 0.09%), given alpha = 0.05 (2-sided), power = 80%, mean of 119 pediatric beds, length of stay = 4 days, with 0.90 average occupancy, k = 0.15, n = 20, baseline rate = 5.1/1000. The quantity k is the coefficient of variation for mortality between hospitals. Assuming attrition of 1-2 hospitals, we will include a sample of 22 hospitals.

Data from 4 pediatric hospitals in Ontario indicate there are 1052 urgent ICU admissions/year, of which we estimate 40% involve SCDE, for a rate of 2 per thousand patient days. Thus, with 20 sites we will be able to show a 31% reduction in SCDE [alpha = 0.05 (2-sided), power = 80%, k = 0.15, where k is the coefficient of variation for SCDE rate between hospitals]. The value of k allows for a moderate amount of clustering within hospital.

Data analysis

Demographic and unadjusted outcomes data will be reported using means, median, variances, interquartile ranges or as proportions with 95% confidence intervals. Outcomes will be reported for the baseline and intervention periods for each hospital consistent with published recommendations for cluster randomized trials [62].

Primary analysis

All-cause hospital mortality will be evaluated using a logit regression model. The dependent variable will be the logit of the proportion dying in each hospital. The independent variables will include a dummy indicator for treatment arm, the baseline mortality logit and the hospital size stratification variable. The analysis will be weighted by the size of the hospital.

Secondary analyses

An identical logit model will be used for ICU mortality after urgent ICU admission where the PICU admission was initiated in a hospital ward, ICU mortality after urgent ICU admission initiated in the OR, unplanned hospital re-admission within 2 days of hospital discharge and mortality following DNR orders.

Significant clinical deterioration events, Code Blue events, stat calls and urgent PICU consultations per 1000 patient-days will be evaluated using Poisson regression using hospital-level aggregated count data. The independent variables will include a dummy indicator for treatment arm, the baseline event rate and hospital size stratification variable. A linear regression model, in which the within-hospital mean is the dependent variable, will be used to evaluate the nature of SCDE, inpatient ward patient-days, and for patients admitted urgently to the PICU, PIM2 score, PELOD score, ventilator-free days, ICU patient-days and ICU therapy-days. The independent variables will include a dummy indicator for treatment arm, the baseline means and hospital size stratification variable. The analysis will be weighted by the size of the hospital.

An identical linear regression model will be used to evaluate the frequency of vital sign documentation from the randomly selected patient records. Since the number of records abstracted will be the same in each hospital, weighting will not be necessary. Documentation will be evaluated as follows: the number of heart rate, respiratory rate, systolic blood pressure, trans-cutaneous oxygen saturation, capillary refill, oxygen therapy and respiratory effort measurements, and the number of times that all seven items were documented will be compared for patients in hospitals with and without BedsidePEWS. All analyses will adjust for baseline event rates and hospital strata. Linear regression will be used to compare the numbers of (1) vital signs documented, (2) documented physician visits, (3) the nurse: patient ratio, the use of (4) continuous ECG monitoring and (5) continuous pulse oximeter monitoring with the BedsidePEWS scores calculated from the abstracted clinical data.

Analysis of the documentation and interaction survey and the post-study decision-maker survey data will compare groups using linear regression weighted by the size of the hospital.

The following four pre-specified subgroup analyses will be performed. (1) Hospital size. Hospitals will be classified on the basis of the number of eligible inpatient ward beds. Hospitals with ≥200 eligible inpatient ward beds will constitute one group and those with <200 eligible inpatient ward beds the other. This is consistent with the stratification method used for randomization. (2) Hospitals with and without MET-RRT. (3) Hospitals with ECMO for children. (4) Patients with urgent PICU admission initiated in an inpatient ward.

Planned conduct

Initial application for funding was made in September 2009, accompanied by 18 letters from hospital CEOs or other senior administrators indicating willingness for their hospital to participate. Successful re-submission in March of 2010 preceded announcement of funding in July 2010. Study resources were obtained from Canadian Institutes of Health Research (CIHR). After approval of $4 million by the reviewing panel, a budget of $3.3 million was granted. Funds were released by the CIHR in December 2010 following recruitment and approval of the three-person data safety and monitoring board, clinical trial registration (CTN#1000018562) and Research Ethics Board (REB-IRB) approval at the sponsoring institution. The study coordinating center is at the Center for Safety Research at the Hospital for Sick Children, Toronto. The study staff is supported by the infrastructure of the Child Health Evaluative Sciences Program of the Research Institute of the Hospital for Sick Children.

EPOCH REB-IRB applications sought and obtained waiver of patient-level consent (Table 6). The local documentation system is used as a part of routine hospital practice, patient-level data are retrospectively obtained and precedent exists in Canada and the UK [51,52]. Furthermore, consent for the participation of 100,000 patents is not feasible, and if required, would significantly undermine the scientific validity of the study. REB-IRB applications also seek approval of implied consent from healthcare professionals for staff surveys and the post-study survey of administrative perspectives [52]. Hospital recruitment though the networks of the CCCTG preceded the enrollment of hospitals with an identified site investigator, administrative and research ethics board approval, and it was contingent on the number of sites previously enrolled in the study. The enrolled sites and associated ethical approvals are provided in Additional file 2.

Table 6 Rationale for waived patient consent in EPOCH
Full size table

Conflict of interest management

There are two parties with conflict of interest that are involved in EPOCH. These are the PI (CSP) and the host institution, The Hospital for Sick Children, who are both shareholders in a computerized decision support company (Bedside Clinical Systems). BedsidePEWS is an FDA-approved product of Bedside Clinical Systems. The study PI (CSP) is a named inventor of the BedsidePEWS (US patent 12/669,896). The patent is owned by the host institution (SickKids).

Study processes to ensure transparency, independence and oversight are in place to manage these conflicts of interest. First, EPOCH employs routine quality assurance processes including Data Safety Monitoring Board Review, annual reporting to CIHR, and scientific and operational oversight from the Executive Steering Committee and the Canadian Critical Care Trials Group (CCCTG).

Second, the PI is separated from key study processes including randomization, allocation concealment, data inspection and the conduct of analyses. The study statistician (AW) is responsible for overseeing the random allocation of sites to intervention or control arms, supervises the project manager and uses a password-protected account on randomizedtrials.net. The password is known only to the project manager and statistician, who complete the randomization process and inform the PI and site investigators of the outcome during the first 2 weeks of data collection at each site.

Third, review of site progress in sites randomized to implementation of the BedsidePEWS is conducted by the Executive Steering Committee in the 4th week of the run-in phase. The decision to complete or prolong the run-in phase is made by the majority vote of the Executive Steering Committee. The PI does not vote.

Fourth, data integrity is ensured by three methods. (1) Separation of data collection from randomization status will be achieved by training site coordinators before randomization, using standardized materials. (2) Inspection of study data during site visits is conducted by the Center for Safety Research staff. If present, the PI will not review primary study data. (3) Following creation of the ‘locked’ data set, each site investigator receives a data report to review, reconcile with the data from their site and provide written confirmation that the data are correct or require reconciliation. Once confirmed by all site investigators the final data set will undergo planned analyses.

Fifth, once completed the analyses will be reviewed by an independent statistician based at another university.

Sixth, the interpretation of the data and their reporting in iterative drafts of the main study manuscript will be a shared activity of the executive steering committee and writing committee and will be complimented by pre-submission review by independent CCCTG reviewers.

Discussion

Preliminary data suggest that the BedsidePEWS score can identify patients at risk for near and actual cardiopulmonary arrest, can be calculated both rapidly and reliably and that its clinical implementation is associated with potential improvements in patient and process of care outcomes. The resource implications of system-wide implementation, the previous lessons from the MERIT study, and the uncertain effect on important clinical outcomes underscore the relevance of the cluster-randomized design, the selection of the intervention and the outcomes used in EPOCH. EPOCH will be conducted in 22 hospitals, involve an estimated 100,000 patients admitted for about 400,000 patient-days and describe approximately 1900 urgent ICU admissions, 200 cardiac arrest events and 500 deaths. Mortality in hospitalized children is an objective measure of quality of care and is used extensively to benchmark the performance of PICUs [58,63-66], surgical programs and hospitals [67-69].

All-cause hospital mortality was selected as the primary outcome in EPOCH for the following reasons. First, mortality is objective and can be reliably assessed. Second, a reduction in all-cause hospital mortality is the ultimate goal of any intervention to improve the outcomes of care of hospitalized children. Third, observational studies of MET-RRT have reported reduced all-cause mortality [36,38]. Fourth, several studies show that when children who have been hospitalized die, most do so in hospital [70-73]. In a 6-year study of in-ICU cardiac arrest, we found only one (1.4%) additional death after index-hospitalization discharge and within 12 months of the index cardiac arrest. Fifth, the effect of pediatric palliative care services on displacing place of death to home or hospice is small [72,74-77]. One large US study found the proportion of deaths in the hospital decreased by less than 6% (from 85.7 to 80.1%) over a 10-year period [75]. Thus, measuring hospital mortality provides a reasonably accurate measure of mortality in children who have been or are currently hospitalized.

As well, all-cause mortality is an established quality metric in Canadian and British adult hospitals and is publicly reported as the standardized mortality ratio in Ontario Hospitals [68,69,78-80]. Finally, while death ‘with DNR’ vs. ‘unexpected’ deaths (no DNR) has been used to distinguish ‘preventable’ from ‘unpreventable’ deaths following acute events in adult patients [34,81], in hospitalized children there are significant limitations to this distinction: (1) DNR orders reflect current expectations of outcome and not the ‘preventability’ of the preceding clinical events; (2) the majority of deaths in hospitalized children, especially in the ICU, occur some days after a clinical deterioration event [3,7,10,18,21,82], but within a relatively short time (hours) between DNR order and death [83-85]. Before confirming that a death in a patient with a DNR order was truly unpreventable, the clinical context and timeline of clinical deterioration prior to the DNR order must be known. It must also be recognized that medically ‘futile’ resuscitation may be performed on hospitalized children [86] for whom no DNR order is in place.

EPOCH outcomes are assessed for 1 year after hospital-wide implementation of intervention or continuation of standard care. This duration was chosen to reduce the effects of seasonal variation, but also to reduce the burden of a multi-year intervention period on each hospital, thus increasing feasibility, and to support timely completion of the study to support the decision-making needs of administrators. Continued eligibility will be confirmed by intermittent reporting of the MET-RRT and Early Warning Score use during the course of the study.

Limitations of EPOCH

EPOCH has some limitations. First, by excluding the use other severity of illness scores, EPOCH will not provide data comparing alternate scores. This more refined question will need to be evaluated separately, preceded by appropriate studies to support selection of best comparators. Evaluation of the ‘best available’ system is important for proof of concept, needed before these more refined questions are addressed. Second, EPOCH does not assess quality of life, parental perceptions or neuro-cognitive outcome. All are relevant measures of the process and outcome of care [11]. In other prospective work we have shown that urgent ICU admission is associated with reduced health-related quality of life and acquired neuro-cognitive deficits [11]; thus, it would be reasonable to include these measures as outcomes in EPOCH. Exclusion of these elements was due to logistic constraints; acquisition of these data for all admissions would not be feasible, and following these urgent ICU admissions would be both resource intense and would require prospective consent/assent and follow-up. Finally, mortality may be both too coarse a measure and too infrequent an event to demonstrate clinically important differences.

EPOCH is a 22-hospital cluster-randomized trial evaluating the impact of the BedsidePEWS as a mechanism to improve the identification of children with evolving critical illness, reduce near and actual cardiopulmonary arrest, and reduce all-cause hospital mortality in hospitalized children. The results of this trial will provide a scientific basis for local, regional, provincial and national decision-making and for recommendations to national and international bodies [87], such as the AHA and European Resuscitation Council [88] about cardiac arrest prevention and institutional best practices.

If negative, the costs of hospital-wide implementation can be avoided, resources more appropriately allocated and the collaborative multi-site data set from the study used to develop and evaluate new hypotheses about cardiac arrest prevention and processes of care. If positive, then EPOCH will have provided a scientific justification for the major system-level changes required for implementation and supported the creation of evidence-based policy.

Trial status

At time of manuscript submission (October 2014) ethical approvals had been obtained at each participating site, study enrolment was complete, data collection had ended in 19 hospitals and was anticipated to be completed by July 2015 in remaining sites.