Introduction

Medical errors simultaneously increase expenses to the healthcare system while decreasing quality of care, particularly in the intensive care unit (ICU). A recent Department of Health and Human Services report highlights the high incidence of medical errors in the intensive care setting in acute care hospitals, resulting in harm to patients and nearly $324 million in monthly costs to the Centers for Medicare & Medicaid Services (CMS) [1]. While some institutions have achieved substantial quality improvement using protocolized bundles and checklists, evidence points to a persistence of errors and poor compliance with best practices in real world care.

The initial findings from The Healthcare Information and Management Systems Society (HIMSS) show a positive impact of electronic medical records (EMR) on clinical outcomes [2]. However, despite high expectations, the implementation of healthcare information technologies (HIT) in the acute care hospital setting has not been shown to consistently decrease errors or reduce costs [3]. In one extremely concerning example, implementation of HIT in a pediatric ICU was associated with a doubling of the mortality rate among critically ill children [4]. The discrete nature of data stored in the EMR, translated directly from the paper environment, can result in fragmented, provider-centered care delivery and impede effective communication. Delayed and poorly contextualized presentation of an enormous amount of rapidly changing data, coupled with information overload, is an important cause of HIT failure in the ICUs [5, 6].

Patient Centered Cloud-based Electronic System: Ambient Warning and Response Evaluation (ProCCESs AWARE) is a novel acute care EMR interface, developed at the Mayo Clinic, Rochester, MN, with funding from a CMS Health Care Innovation Award. AWARE contains built-in tools for error prevention, practice surveillance, decision support and reporting, built on an advanced understanding of cognitive and organizational ergonomics. In preliminary studies this system has significantly decreased the cognitive load of bedside clinicians, reduced medical errors [6], and was associated with improved communication, care efficiency, and ease of clinical data management compared to the standard EMR [7].

The aim of this study is to quantify the impact of AWARE on a range of care process and patient health outcome metrics. We compare outcomes of patients through several levels of adoption of AWARE to historical controls pre-AWARE implementation.

Methods

Study design and enrollment

We performed a comparison before and after ProCCESs AWARE implementation using a prospective cohort from 2014, compared to a historical control. The study enrolled all critically ill adult patients (over 18 years old) in four adult ICUs at Saint Mary’s and Methodist campuses of Mayo Clinic Hospital in Mayo Clinic, Rochester MN, who stayed in the hospital at least 24 h. The control group was drawn from 2010, in order to avoid contamination effects of the use of early versions of the AWARE system, which were introduced during 2011–2013.

Intervention

AWARE is an ICU-specific patient viewer/monitoring system based on. NET technology, a standard software framework developed by Microsoft. The routine use of AWARE in clinical practice started in July 2012 in the medical ICU at Mayo Clinic Rochester (MN) [7]. The complex system of ranks and decision rules allow AWARE to extract and to organize relevant patient data in comprehensive patient-oriented information blocks. User interface provides patient identification, clinical data, administrative information, and task-specific views within patient-specific boxes displayed in the context of a global overview of the physical layout of ICU beds. Each box can be selected to expand patient-specific information such as the clinical problem list [8], procedures, and detailed (notes, procedures, labs, tests, medications) organ system information. Seven color-coded pictograms (central nervous system, cardiovascular, respiratory, renal, gastrointestinal, blood & coagulation, and infection status) represent clinical significance of the current problem. (Supplemental Digital Content - Figures S1 and S2). Patient-specific checklist compliance and decision support tools are available for each patient. Shared electronic task lists are used for communication and hand off between team members [9]. Smart alerts and reminders are integrated with predefined escalation path. Administrative dashboard shows statistics on the user interaction with the system, such as checklist compliance and decision support tool utilization rates (Supplemental Digital Content - Figures S3 and S4).

While the use of AWARE was not mandatory to all providers, it was encouraged by local champions and critical care leadership. Access to this system was available at every hospital’s workstation and through mobile devices. All clinical personnel including nurses, nurse practitioners, physician assistants, respiratory therapists, and physicians were able to use the system during morning rounds and at any other time during the patients’ ICU stay.

AWARE adoption levels were defined apriori using the percentage of patients with completed checklists of best practices of care: high >80% and moderate >50%.

The Institutional Review Board reviewed and approved this minimal risk study, and granted a waiver of informed consent for data collection.

Data collection

Data were extracted from the AWARE system and hospital systems, including electronic medical records and administrative information systems, using previously developed real time informatics infrastructure for syndrome surveillance, decision support, reporting, and modeling of critical illness [10]. These data are used for quality report generation and decision support systems and were found to be consistent throughout the study period [11, 12].

Financial data were adjusted for inflation using the Consumer Price Index and reported in 2014 US dollars.

Outcome measures

We analyzed data characterizing care process quality and patient health outcomes.

Health outcomes evaluated included: overall and ICU length of stay (LOS), inpatient and ICU mortality, central line infections, urinary catheter infections, and ventilator associated events (VAE). Centers for Disease Control/National Healthcare Safety Network (CDC/NHSN) VAE definition was used [13, 14]. In addition we evaluated the overall cost of hospitalization in terms of hospital charges.

The main process outcomes in this study included resource utilization in terms of days with central line usage, days with urinary catheter usage, days of antibiotic usage, days on continuous intravenous (IV) sedation, and days on mechanical ventilation. These measures were detailed during the study ICU stay as utilization days per ICU day. Ventilation days were measured per entire hospital stay. For patients on mechanical ventilation, we assessed compliance with lung-protective mechanical ventilation best practices based on Acute Respiratory Distress Syndrome Network (ARDSNet) definition [15], and reported as average and median % of daily compliance per patient on ventilation. In addition we assessed compliance with the daily ventilator bundle, a best practice guideline for care of ventilated patients [16]. The ventilator bundle included deep vein thrombosis (DVT) prophylaxis, gastrointestinal (GI) prophylaxis, sedation holidays, assessment of readiness to extubate, and elevation of the head of the bed. This measure was reported as average and median % of daily compliance per patient on ventilation.

Additional process outcomes in this study included documentation for a number of best practices of care. We measured documented assessment of central line removal (mean number of assessments per days with central line), documented assessment of antimicrobial discontinuation (mean number per days on antimicrobials), documented family conference (mean number of conferences per patient ICU stay), and palliative care offered (% of patients).

Statistical methods

We examined the difference in process and outcome metrics from baseline data collected prior to AWARE implementation to those measures collected subsequent AWARE implementation using non-parametric Wilcoxon tests for continuous characteristics and chi-square tests for categorical measures. We use a significance level of p = 0.05.

Due to the observational nature of this study, covariate adjustments were made to control for known biases and confounding. The adjustments included patient acuity and comorbidities using APACHE III scores, patient age, gender, and ICU. We used spline adjustment on the continuous measures of age and APACHE III score (measured at 24 h after admission) to control for non-linearity. The analysis of financial data (charges) was also adjusted by patient surgical vs medical status. In order to test for the effect of the intensity of adoption as a “dose-response effect”, we also ran models for the mortality and LOS measures, adjusting for level of adoption at the study ICU as a binary variable.

Mortality measures were modeled using logistic regressions. Skewed measures, such as length of stay in the hospital or the ICU and the charges for study hospitalization, were modeled using linear regressions with log-adjusted dependent variable. Generalized linear models were used to model changes in event rates using Poisson outcome distributions.

All analyses used the SAS software version 9.4 (SAS Institute Inc., Cary, NC, USA).

Results

The baseline demographics are described in Table 1. The patients in the study group were slightly younger with a slightly higher proportion male (59% vs 55%), and similar racial/ethnic profiles(about 91% white). The mean APACHE scores were higher in the study group, 62.8 compared with 49.1 (p < 0.0001). The proportion of patients on mechanical ventilation was lower in the study group, 40% compared with 54%.

Table 1 Cohort demographics and baseline characteristics
Full size table

Table 2 shows the unadjusted rates of outcome measures. There was a lower overall and ICU mortality (from 7.2% to 6.1% and from 4.6% to 3.4%, respectively), but the differences did not reach statistical significance. There were statistically significant reductions in overall and ICU LOS (from 12.1 to 8.2 days and from 4.1 to 2.5 days, respectively), and in total charges (from $149,593 to $103,383 average per hospitalization). There were significant differences in the number of days with central line, urinary catheter, continuous IV sedation, number of patients receiving invasive ventilation and duration of mechanical ventilation, and the use of antibiotics decreased from the control group to study group. In fact the proportion of patients with any central line insertion, urinary catheter insertion, sedation, ventilation, or the use of antibiotics decreased.

Table 2 Unadjusted process and health outcomes metrics, pre- and post-aware implementation
Full size table

Table 3 shows the results of the unadjusted and adjusted The odds of death after adjusting for APACHE score and other relevant confounders was much lower in the study group. Overall in-hospital and ICU mortality unadjusted odds ratios for the AWARE intervention were 0.83 and 0.73 (p = 0.33 and 0.2, respectively). After adjusting for patient acuity (APACHE score) and baseline demographics, the odds ratios were 0.45 and 0.38 (p < 0.001). The ICU length of stay decreased by about 50%, the hospital length of stay by 37%, and the total charges for hospital stay by 30% in post AWARE cohort (by $43,745 after adjusting for patient acuity and demographics).

Table 3 General linear models for the effect of AWARE intervention, unadjusted and adjusted by patient characteristics
Full size table

The analyses adjusting by level of adoption had similar results to results in Table 3. After adjusting for patient acuity (APACHE score), baseline demographics, and high level of AWARE use, the odds ratios for hospital and ICU mortality in the study group versus the comparator group were 0.48 and 0.39 (p < 0.001), respectively. Fig. 1 shows odds ratios for hospital mortality by ICU. The ICU and hospital LOS decreases were the same with this adjustment as those shown in the adjusted analysis in Table 3, p < 0.001. In each case high use ICU was significantly associated with improved outcomes compared to moderate use ICU (p < .0001, p = 0.003, and p = 0.04, respectively), except for hospital LOS (p = 0.2).

Fig. 1
figure 1

The odds ratios of in-hospital mortality with and without the AWARE intervention, stratified by study ICU, with multivariate adjustment. Note: The overall p-value for the unit*intervention interaction was 0.33

Full size image

In the study group there were reductions in central line days per day in the ICU (45% reduction to 0.3), urinary catheter days per ICU day (13% to 0.7), days with antibiotic use per ICU day (18% to 0.6), days on continuous IV sedation per ICU day (30% to 0.3), days on invasive ventilation (50% when adjusting for patient characteristics). The rates of documented assessment of central line removal and antibiotic discontinuation were lower by 15% and 57%, respectively. For patients on invasive ventilation, compliance with lung protective ventilation thresholds of tidal volume improved by 73% (up to 92% compliance), but there was no statistically significant effect on the rates of compliance with the ventilator bundle. We also observed a statistically significant increase in the use of family conferences per ICU day (26% increase). Rates of use of palliative care did not change.

Discussion

In our before and after comparison study we evaluated the effect of implementation of electronic system specifically designed to improve the patient information integration and provide decision support for clinicians in the ICU, and we found significant differences in measured outcomes. Reductions in both hospital and ICU mortality, length of stay, and costs of hospitalization were significant when controlled for patient characteristics and AWARE usage levels.

The process measures showed mixed results. A large difference in the number of patients receiving MV may be explained by the increased use of non-invasive ventilation in recent years as a first line treatment in appropriate cases comparing to 2010. The use of central lines, urinary catheters, antibiotics and continuous IV sedation decreased, a positive effect as these factors may increase risks of adverse outcomes in patients [17, 18]. The increase in days of antibiotics per ICU day may reflect the decreased total number of days in the ICU. Due to the decreased length of stay, process outcomes calculated per day of ICU (such as antibiotic utilization) did not appear to improve.

Rates of compliance with lung-protective mechanical ventilation best practices improved, although rates of documented compliance with the ventilator bundle did not change. The use of family conferences increased, perhaps indicating more time spent by clinicians communicating with patients and families. On the other hand, the rates of documented assessments for central line removal and antibiotic discontinuation decreased, possibly reflecting a decrease in rates of documentation rather than actual assessment performed by clinicians, or the use of checklists which also include assessment to discontinue antibiotics. We were not able to directly measure behavior related to these assessments.

Two main ideas could explain the association of AWARE with improvements in patient care. First, AWARE may directly impact providers’ provision of care by more ergonomically providing healthcare providers the information influential for clinical decision making. Second, AWARE may indirectly impact patient care by allowing providers to more easily treat more patients, producing a spillover effect by releasing time, which can be used to benefit the overall census of the ICU. Improved care processes should in turn improve immediate health outcomes. Furthermore, AWARE allows ICU directors and manager nurses to control whether a built-in process checklist is used, a feature that was not available in the EMR system prior to the implementation of AWARE. Checklists for processes of care provide guidance to improve adherence to best practice [19], which may also lead to better outcomes.

Recent experience with a real-time safety-bundle dashboard, implemented in one of pediatric ICU and focused on a process of care and resource utilization, reports the increase of awareness for potential interventions, as well as modulation of outcomes [20]. However previous studies failed to demonstrate a sustainable improvement in mortality or ICU LOS. One study provides evidence for small but significant changes in reduction of length of stay and 30-day mortality but no changed in inpatient mortality with the introduction of a basic EMR in US hospitals [21].

Limitations

The difficulties with evaluation of the effect of electronic interventions such as AWARE, and other systems interventions, are associated with controlling for confounders and isolating the treatment effect of the intervention. Unfortunately, this is a common problem with technology evaluation in real world scenarios as these interventions cannot be implemented in a traditional randomized controlled trial. Before-after comparison study, while lacking the power of randomization, is widely used and typically the most logistically feasible and practical study design.

Patients in the post-AWARE group appear sicker compared to pre-AWARE group.. Both comparison groups were drawn from the same ICUs, and analysis controlled for the patient’s ICU in order to remove confounding due to the differences in the types of patients by unit, i.e. medical, surgical, cardiovascular, and other patients. The analysis also controlled for patient severity, using the APACHE score as a covariate.

A common criticism of before/after comparison studies is that changes in outcome measures may be attributed to other changes or quality improvement initiatives that may have occurred over time. Unfortunately it is difficult to assess such impact without availability of a concurrent control group. Instead we attempted to examine the effect of the intervention on outcomes by the usage level of the AWARE intervention. When we analyzed outcomes by whether the patient was in a high AWARE use unit compared to moderate use AWARE unit, the association between AWARE and improved outcomes was preserved, but unfortunately the study was not powered to show a dose-response effect.

The VAE surveillance definition algorithm was developed and implemented in the NHSN in January 2013 [22]. Pre-AWARE cohort predates implementation of this metric; therefore, the numbers may be underestimated in the control group.

Since clinicians were free to use AWARE or standard EMR, the “efficacy” of intervention could be underestimated. However, a parallel study showed that about ¾ of clinicians intended to use AWARE after its implementation [23].

Conclusions

Better organization of information in the ICU with systems like AWARE has the potential to improve important patient outcomes, such as mortality and length of stay. By reducing the length of stay in the hospital as well as ICU, the costs of hospitalization are also greatly reduced. Objectively measurable process outcomes, such as use of central lines or continuous IV sedation, also improved in accordance to best practices of care. However, rates of documentation of care processes are not necessarily affected by the system. While there is much variation among HIT systems and their effect on care and outcomes, a carefully designed system holds great promise to improve health and care quality and efficiency. However, due to study design limitations we cannot prove causality, and the magnitude of the main outcomes in this study should be interpreted with caution.