1 Background

Variability in nature is a central property of the biosphere. In Darwinian evolution natural selection operates on organisms with genetic variation to favor those best matched to the local environment [1]. In contrast, natural variation in clinical decision-making is linked to cultural (Lamarckian) inheritance, garnered predominantly from clinical training [2, 3]. Variation in clinical decision-making is a resource because it exposes multiple options of response to clinical challenges. Some clinical decisions may be clearly associated with better outcomes and lead rapidly to improved care. Unfortunately, the advantages and disadvantages of different clinical decisions are seldom clear to clinicians because: (1) healthcare processes are complex, (2) many variables in addition to a clinical decision influence the clinical outcomes of interest, and (3) different treatment options usually have small effects on important clinical outcomes [4]. Variable clinical decisions are likely to be passed on to trainees who perpetuate the variations without being able to evaluate whether one approach is better than another.

When the impact of an intervention or clinical decision on a clinical outcome is large (e.g. appendectomy for acute appendicitis), the signal is clear and the medical community rapidly adopts the preferable option. Thus when the signal is large, the medical community acts in a way analogous to natural selection in Darwinian evolution. Unfortunately, the impacts of many interventions or clinical practice processes on clinically important outcomes are small, and remain hidden from clinicians [4]. Clinical outcomes frequently require large clinical trials for identification of small signals [4]. In addition, clinical practice processes (cointerventions in clinical trials) can influence the impact of a specific intervention under study, and may obscure or reverse the true clinical trial result [5]. Therefore, natural variation in clinical decisions (clinical practice processes), without systematic study, can confound clinical trial results. For much of clinical medicine there is no obvious signal, and there is no natural selection toward the best-suited option. Much of clinical medicine, then, does not naturally evolve to improve healthcare quality.

Control of variability (stabilization of process) is central to process improvement in many human endeavors [6–8] and forms the foundation of healthcare quality improvement [9]. After stabilizing the process, we can systematically study whether one approach is superior to another using rigorous scientific methods [5]. We have demonstrated the feasibility of stabilizing the clinical decision-making process with electronic decision support tools (eProtocols). We use the term eProtocols to designate explicit, complex, iterative decision support protocols, as opposed to simpler and more common forms of clinical decision support, such as alerts for abnormal lab values. The eProtocols are typically implemented in computer software to facilitate accurate and consistent navigation of the protocol, to reduce protocol complexity by hiding detail until needed, and to facilitate data capture regarding protocol usage. We have used eProtocols to improve healthcare quality [10–16]. These results refute the objections of some clinicians. In response to those who cite Emerson “A foolish consistency is the hobgoblin of little minds,” evidence indicates that stabilization of clinical process is a wise consistency. We believe this consistency can clarify many issues in medicine, including the impact of decision support tools themselves [17].

In this manuscript, we describe our 26-year experience developing eProtocols that enable reproducible clinical research and care methods. We began development of eProtocols in the 1970s to standardize clinical decision-making in the pulmonary function laboratory [18–20]. Our unique, supportive local environment and goal alignment between the clinicians and administrators enabled local development of eProtocols before extensive data regarding eProtocol effectiveness and outcomes were available. We began with a hard-coded mainframe computer program for mechanical ventilation [10–12]. During the last two decades, our eProtocols have evolved to stand-alone applications, and then to web-based applications available in multiple languages (Fig. 1 and 2, Table 1). Next, we examine eProtocol-insulin development and implementation as an example of protocol evolution. We outline challenges that remain in the development and use of eProtocols, especially in the areas of technological standardization, regulation and oversight, and eProtocol maintenance. Based on our experience developing and validating multiple eProtocols (for management of mechanical ventilation, intravenous fluid, and blood glucose), we offer an eProtocol development roadmap that may be useful for developers and clinicians interested in clinical decision support (see below, Table 2).

Fig. 1
figure 1

Temporal development of eProtocols with different computer platforms

Full size image
Fig. 2
figure 2

eProtocols under current development

Full size image
Table 1 Attributes of different eProtocol versions
Full size table
Table 2 Proposed eProtocol development process
Full size table

2 Local environment

A successful change in clinical process requires environmental support. The local group at LDS hospital (formerly the Latter-Day Saints Hospital) enjoyed a unique environment of collaboration that enabled us to invest the time and effort necessary for protocol development. We believe institutional backing and frontline clinician support are both important elements for successful eProtocol development. We began our first knowledge engineering efforts in 1985 for eProtocol development (Fig. 1, Table 1). We had an invested clinician leader (AHM), and a shared commitment to eProtocol development by the local research and clinical practitioner groups. This level of commitment from the group, from the institution and its practicing clinicians, allowed the initial knowledge engineering sessions to include many of the staff intensivists, ICU nurses, pharmacists, and informaticists. For example, our initial logic development meeting had fourteen participants. The participants included pulmonologists, anesthesiologists, private practice intensivists from LDS Hospital, academic intensivists from the University of Utah, the LDS Hospital, and the University of Milan (Italy), critical care nurses, a critical care respiratory therapist, and a medical informatics graduate student.

Our initial eProtocol effort was new and therefore much discussion and collaboration was required to build the necessary trust for clinical application. When we began there were no clinical outcome data and we were not sure detailed computer protocols would be feasible in a clinical setting. Intense participation by clinicians enabled broad acceptance of the eProtocol and its successful clinical implementation. We used the best evidence in the published literature to establish eProtocol rules. We used randomized, controlled, clinical trials published in respected journals and established physiologic relationships. When such evidence was unavailable for required eProtocol rules, we used discussion to reach a consensus of expert opinion (a modified Delphi technique). We continued discussions until we agreed on a reasonable strategy and set of rules. We emphasize this central point. In the absence of evidence, one must adopt a reasonable strategy. This agreement on a strategy stabilizes the decision-making process [5] and provides a method of care that can be used as a comparator for testing proposed changes, and for making progress through iterative refinement. We established a process for eProtocol development and its important developmental steps, that includes iterative refinement and validation of eProtocols [22]. Iterative refinement is necessary because we are not able to anticipate all the contextual challenges that will present themselves to the eProtocol when used in a clinical environment.

We used the LDS Hospital environment as a human outcomes research laboratory for our quality improvement eProtocol work. In addition to the delivery of good and ethical care, LDS hospital had two attributes required of any laboratory: reliable data capture (we have had a functional electronic medical record (EMR) since 1972) and reproducible methods (eProtocols). These attributes have enabled our iterative refinement of eProtocols during the past several decades.

3 Local development

We developed one of our first eProtocols to manage mechanical ventilation (eProtocol-MechanicalVentilation) in a randomized clinical trial of extracorporeal support in lung failure patients [11]. We recognized the importance of decreasing between-group variability in mechanical ventilator support for subjects in the clinical trial. We chose to model the clinician decision-maker, rather than model the patient [5]. We used a production rule (if–then) strategy. We embedded the production rule eProtocol-MechanicalVentilation in our HELP-1 mainframe computer system [23, 24]. Encouraged by our experience with eProtocol-MechanicalVentilation, we then studied its performance (compared to usual care) in a multi-center, randomized, clinical trial. The multi-center trial required that we transfer the eProtocol-MechanicalVentilation production rules from the Intermountain HELP-1 mainframe computer system to a personal computer (PC) [10, 12]. eProtocol-MechanicalVentilation was associated with shorter ventilator weaning times and less hypoxemia. As a result, one of the participating sites continued to use eProtocol-MechanicalVentilation in their clinical care. Thus, they improved care and controlled this cointervention for their subsequent research purposes [25, 26].

We subsequently developed PC stand-alone Visual Basic 6 (Microsoft, Seattle, WA) versions of eProtocols for multi-center randomized, controlled trials of the National Institutes of Health (NIH)/National Heart, Lung and Blood Institute (NIH/NHLBI) Acute Respiratory Distress Syndrome Network. These Visual Basic 6 eProtocols were used for management of both mechanical ventilation (eProtocol-MechanicalVentilation) [27] and intravenous fluid (eProtocol-Fluid) [28, 29].

4 Evolution of eProtocol-insulin: an example of eProtocol development, validation, and exportation

As an example of the evolution of eProtocols, we focus on eProtocol-insulin. We initially developed eProtocol-insulin [24–27] at LDS Hospital, following a report that control of hyperglycemia conferred a survival advantage [30]. The initial eProtocol-insulin was a quality improvement effort intended to manage stress hyperglycemia in the adult ICU.

We initially developed and validated a stand-alone eProtocol-insulin version (Visual Basic 6) at a single site in the United States (LDS Hospital) with new knowledge engineering tools [31] (Figs. 1, 3, Table 1). We captured the reasons for which clinicians declined eProtocol recommendations. Bedside clinicians selected these reasons from a drop-down menu in eProtocol or typed free-text reasons for declining recommendations. These reasons were then analyzed and contributed to the iterative refinement of the rules [15]. We conducted focus groups to evaluate the user interface using standard decision support software validation and verification techniques [32]. We retested and revalidated eProtocol-insulin after every substantive change in the eProtocol. We assessed blood glucose values in ICU patients pre- and post- eProtocol-insulin use. We validated local LDS Hospital clinician acceptance (~94 % of eProtocol recommendations), and safety (severe hypoglycemia, defined as serum or blood glucose ≤40 mg/dL (2.2 mM), occurred in only 0.08 % of measurements).

Fig. 3
figure 3

Stand-alone (Visual Basic 6) eProtocol-insulin bedside protocol screen. All the necessary data and eProtocol-insulin recommendations fit on a single screen. Bedside ICU nurses find the large count-down timer (#1) easy to use. It allows timely management of the different interventions (change of insulin drip rate, last glucose measurement, rate of glucose change). The eProtocol-insulin recommendation is in the center of the screen (#2) and the bedside clinician can choose to accept or decline this recommendation (#3). The reason for declining a recommendation is captured, and is part of the iterative refinement process of eProtocol-insulin. The data entry fields for the bedside clinician (#4) include the last blood glucose, the rate of insulin drip infusion, and whether the patient is receiving glucose calories

Full size image

We subsequently translated our eProtocol-insulin research and quality improvement results at LDS Hospital into usual clinical care at multiple Intermountain Healthcare ICUs. Intermountain Healthcare is LDS Hospital’s parent healthcare system and includes 22 hospitals. The eProtocol-insulin rules were embedded in the HELP-2 Intermountain Healthcare electronic medical record system using a Java rules engine called Foresight [33]. This translation was necessary because we developed the initial version of eProtocol-insulin on a platform designed for rapid prototyping and rules development (Visual Basic 6). Each rule was manually rebuilt because of three major incompatibilities between the prototyping platform and Foresight: data model, terminology, and knowledge (rule) representation. Once translated, tested, and validated, both in silico and clinically at the LDS Hospital, the clinical Foresight version of the protocol became available to all Intermountain inpatient facilities where HELP-2 is installed [14].

We continued to refine eProtocol-insulin rules with clinician investigators at eight academic medical centers in the United States [16]. These investigators raised a number of clinically pertinent concerns. They needed assurance that due diligence regarding the safety of eProtocol-insulin had been pursued. We addressed these concerns through discussion, in silico test results review, clinical data review, consensus generation, and finally extensive clinical application and review of results. Clinician investigators met during multiple sessions in person and by teleconference to assess and refine the protocol logic. We examined eProtocol-insulin and its performance and reviewed detailed case histories of patients supported with the protocol. We compared the instructions of eProtocol-insulin to the instructions given by six experienced intensivists. Although most eProtocol-insulin instructions were within the range chosen by the experienced intensivists, those that differed provided important insights. Input from multiple, widely dispersed, institutions introduced different clinical contexts and raised many important considerations.

For example, some clinician investigators were concerned that hypoglycemia would occur if blood glucose were decreased too rapidly. “Too rapidly” was explicitly defined by the group through consensus choice of specific rates of change of glucose over time. In addition, the group was concerned about hypoglycemia after a low blood glucose value that approached hypoglycemia. To address these concerns, we decreased the interval to the next blood glucose measurement from 2 to 1 h when the blood glucose values were below the target range.

Next, we conducted a tightly supervised clinical validation of eProtocol-insulin at the eight academic medical centers. We used frequency distributions of blood glucose to evaluate the impact of eProtocol-insulin on the achievement of target values, time to first measurement within target, and rates of hyper- and hypoglycemia. After safety was clearly established during these refinement and validation cycles by clinicians at LDS Hospital and the eight academic sites, we placed clinically validated, stand-alone eProtocol-insulin applications on laptop computers. We distributed these computers to the academic sites and used the stand-alone eProtocol-insulin in the US (Baystate Medical Center, Tufts University, University of Virginia Hospital and Wake Forest Baptist Health) as well as at the National University Hospital of Singapore [21]. We used an a priori hypoglycemia safety limit of 0.5 % of blood glucose measurements [16]. We validated multiple institution clinician acceptance (92–98 % of eProtocol recommendations), and safety (rate of severe hypoglycemia was only 0.3 % of measurements) [21].

After progress in the adult ICUs, we proceeded with refinement of eProtocol-insulin rules to accommodate the pediatric ICU population [34, 35]. We increased the computation granularity to return insulin dose recommendations in insulin units per kilogram body weight per hour when the “pediatric” mode was selected. We removed the initial insulin bolus dosing. We calculated dextrose bolus doses per kilogram used to treat hypoglycemia for 10 % dextrose and 20 % dextrose solutions. We also reduced the desired rate of change in insulin dose and interval between blood glucose measurements as blood glucose values neared the target range. We compared eProtocol-insulin instructions to the instructions given by five experienced pediatric intensivists. Using the techniques described above, we again validated and verified the knowledge base and software [16]. Clinical evaluation revealed clinicians were unwilling to choose a single blood glucose target therefore, initial validation of pediatric eProtocol-insulin allowed the clinician to choose a blood glucose target for each patient. After using eProtocol-insulin on seven patients with individually chosen targets, clinicians were more willing to accept a single target. We thereafter allowed clinical users to modify blood glucose targets to accommodate subsequent published results (see “eProtocol maintenance” below) [36, 37].

5 Web-based eProtocol with improved displays and usability

Using stand-alone eProtocol-insulin with laptop computers in multiple institutions involved all the problems associated with multiple copies of a computer application on multiple hardware devices. Maintenance was a challenge. We wished to facilitate use of eProtocol-insulin by multiple institutions and developed a web-based version of eProtocol-insulin (and other eProtocols), using an ASP.NET platform. Users enter patient data with a web browser and eProtocol-insulin returns patient-specific recommendations to the clinician. Web-based eProtocol-insulin highlighted technical and regulatory challenges. Institutional policies regarding internet firewalls and patient privacy impeded web-based eProtocol-insulin use. One approach to address these concerns is sending de-identified data to the web-based rules engine that generates a recommendation without retaining any clinical data on the server (e.g. virtual medical record, see Future challenges below) [38].

The transition to web-based eProtocols was coupled with more focus on usability. We had conducted several usability assessments of our stand-alone eProtocols throughout the development and refinement process, through heuristic evaluations with clinical and informatics experts, user focus groups, and formal usability evaluations. We intended to render the web-based version of eProtocol-insulin usable by a clinician without specific eProtocol training. Our long-term focus on usability led to the design of a single interface screen. Figure 4 illustrates the single web-based eProtocol-insulin interface.

Fig. 4
figure 4

Web-based (ASP.NET) eProtocol-insulin bedside screen. eProtocol-insulin bedside screen with count-down timer (#1) indicating 1 hour 43 minutes and 09 seconds remaining before the next mandated blood glucose measurement. Bedside clinicians enter the last serum glucose, the current insulin drip rate, whether the patient is receiving calories and whether an intravenous infusion containing dextrose is present (and its rate) (#2). The date and time fields (#3) are open to allow back-charting of data in between eProtocol-insulin assessments, thus allowing bedside clinicians some flexibility in adding information known to them but not to eProtocol-insulin. After the data are entered, the clinician can press the “Enter Data to get Recommendation” button (#4) to view the eProtocol-insulin recommendations. The orange “Help” button (#5) displays a help screen with five to eight simple steps (explanations) of how to use eProtocol-insulin

Full size image

The single interface contains one orange “Help” button (Fig. 4, #5). Clicking this “Help” button displays a help screen with five to eight simple steps (explanations). All of our web-based eProtocol interface and help screens are similar. The eProtocol-MechanicalVentilation displays illustrate the interface and help screens for a more complicated eProtocol than eProtocol-insulin (Figs. 5, 6).

Fig. 5
figure 5

Web-based (ASP.NET) eProtocol-MechanicalVentilation bedside screen. The eProtocol-MechanicalVentilation screen data fields are populated either by the eProtocol pulling data from the EMR or by the bedside clinician. Clinicians can choose the protocol for sea-level or higher altitude ventilation (e.g., Denver or Salt Lake City) (#1) and the recommendations are found at the bottom of the single screen (#2). Each recommendation can be individually accepted or declined by the bedside clinician (#3), and the reasons for declining a recommendation are captured as part of the iterative validation and refinement process of eProtocols

Full size image
Fig. 6
figure 6

Web-based (ASP.NET) eProtocol-MechanicalVentilation bedside help screen. This single screen is the entire instruction material for the clinical eProtocol user. Each step on the help screen gives a brief instruction for the eProtocol user action. A clinician with no prior eProtocol-MechanicalVentilation experience is very likely to be able to use eProtocol-MechanicalVentilation, without special training, by following these simple instructions given on the help screen

Full size image

6 eProtocol maintenance

We regularly update our eProtocols with the goal of reflecting the most up-to-date published evidence. This knowledge maintenance process relies on the interested individual clinician. The clinicians in charge of a specific eProtocol incorporate new data as they are published. For example, when new studies regarding glycemic control in the ICU were published, eProtocol-insulin was updated to reflect the new data [36, 37, 39–41]. By the time these studies were published, our cumulative experience using eProtocol was ample and we had significant data showing excellent performance and safety of eProtocol-insulin. This institutional experience allowed greater clinician trust, and further updating of eProtocol-insulin neither necessitated discussions among, nor time commitments from, large numbers of clinicians. Different interpretations of the studies regarding glycemic control in the ICU were reconciled more rapidly than in the past through brief meetings. For example, in our current iterations of eProtocol-insulin, users can change the blood glucose target from the original 80–110 mg/dL (4.4–6.2 mM) to their current clinical care targets to incorporate the results from NICE-SUGAR and other studies [36, 37].

7 Future challenges

Despite the significant advances in eProtocol development and use over the past 26 years, challenges to wider use and acceptance of eProtocols remain.

We need to:

  1. 1.

    Avoid double data entry

    1. a.

      link to EMRs and

    2. b.

      establish technological standardization

  2. 2.

    Resolve regulatory issues regarding healthcare software use and development, and

  3. 3.

    Define and standardize processes for eProtocol

    1. a.

      development,

    2. b.

      validation,

    3. c.

      maintenance, and

    4. d.

      storage

  4. 4.

    Coordinate multiple simultaneously used eProtocols according to clinical importance and workflow

  5. 5.

    Reform professional society guidelines to be issued in executable form via eProtocols

7.1 Avoid double data entry

7.1.1 Link to electronic medical record

In clinical research, double data entry is necessary, once for the clinical database and once for the research database. However, such double data entry is a barrier to incorporating eProtocols into clinical practice. It would be ideal if data were entered once, and then could flow both to the EMR and to the eProtocol.

In order to eliminate double data entry at Intermountain Healthcare and to test the feasibility of using eProtocol-insulin to translate research results into clinical practice, we linked eProtocol-insulin with Intermountain Healthcare’s EMR in two ways. First, we installed the stand-alone Visual basic 6 eProtocol-insulin on bedside clinical terminals and interfaced it with the HELP-1 EMR (Fig. 1, Table 1). Next, we embedded the eProtocol-insulin rules in the newer HELP-2 EMR [27]. We mapped serum glucose and the other variables in our local EMR to their representations in eProtocol-insulin. eProtocol-insulin pulled clinically required patient data from the EMR and required no extra data entry by the bedside clinicians. This streamlined information flow avoided additional errors associated with manual double data entry, and it contributed to the high acceptance and use of eProtocol-insulin. The results from use in usual care were almost indistinguishable from the results in the research environment [14].

7.1.2 Establish technological standardization

There is, as yet, no standard terminology and data model that has been uniformly adopted by healthcare institutions. As a result, the work done to link eProtocol-insulin with the Intermountain Healthcare EMR is not directly exportable to different EMRs at other institutions. Achieving interoperability and compatibility standards in healthcare is a long-term goal (viz: The US 2009 Health Information Technology for Economic and Clinical Health Act (HITECH); the Healthcare Information Technology Standards Panel (www.HITSP.org)). We are, however, far from achieving this interoperability goal.

The need to share medical decision support tools is large and growing. Numerous healthcare issues could benefit from decision support. No single institution can address this challenge satisfactorily. Sharing validated decision support tools will require easy access from multiple institutions, advances in mapping to local EMR data elements, ability to move clinical data across firewall boundaries for decision support applications, and likely web-based services, at least for the foreseeable future. The virtual medical record may enable easier inter-institutional access [38].

7.2 Resolve regulatory issues

The US 2009 Health Information Technology for Economic and Clinical Health Act (HITECH) provided unprecedented incentives for adoption and support of EMRs. To qualify for those incentives, clinicians must demonstrate use of technology, such as clinical decision support [42]. These incentives brought increasing interest in use of electronic decision support tools. The increased attention and focus has come with increased interest in regulation.

The increased regulation is a major change from the policy adopted by Frank Young, the US Food and Drug Administration’s (FDA) Commissioner in 1987 [43]. Dr. Young argued then, that the FDA had no role to play when decision support was provided to clinicians in an open-loop manner that required clinicians to accept or decline a protocol recommendation. We use eProtocols in an open-loop manner.

Current FDA proposals identify electronic decision support software and mobile applications (apps) as medical devices that fall within FDA regulatory purview. The implications of this FDA position are not yet clear. While the FDA provides valuable oversight [44], the current position being considered by the FDA now raises serious concerns about impediments to rapid revision and refinement of electronic decision support tools [45]. The emphasis of the FDA on device hazards may make a balanced risk versus benefit assessment for clinical decision support applications difficult [46]. While no one could reasonably dispute that electronic decision support tools should be designed in a manner that promotes patient safety and quality of care, the potential oversight of FDA regulations can impede development and maintenance of computer-based protocols. The wide variety of electronic decision support tools, lack of clarity in definitions of levels of patient risk, and even in definitions of what constitutes a clinical decision support application, complicate FDA oversight [47]. A proposal by the medical informatics community to retain oversight locally [48] seems preferable to us, although external oversight in a manner similar to that provided by the National Transportation Safety Board (NTSB) may be necessary as the number and complexity of these computer-based interventions proliferates [49].

7.3 Define and standardize eProtocol development, validation, maintenance and storage

Defining a standardized process for protocol development and validation would be an important step in broadening the use and acceptance of eProtocols. Such a process may facilitate the regulatory oversight of eProtocols and ensure patient safety. Based on our experience, we propose the steps in Table 2 for eProtocol development. We believe all eProtocol development should start with a clear definition of the goal, the outcome measure of interest, and the key team members who will participate in knowledge engineering.

The key team members of the knowledge engineering group should include expert clinicians, a clinician champion, knowledge engineers (who can navigate some intersection of informatics and clinical content), and all stakeholders, including ancillary staff and clinicians. A broader group of participants should also be identified—the clinical review group. This group need not be present for all knowledge engineering sessions, but will review and test the eProtocol and provide additional feedback (e.g. beta testing). eProtocol logic rules can be communicated among different team members with flow diagrams, tables, or other written communication.

Development of the protocol logic starts with a literature review. Much of the logic in eProtocols is not based on randomized controlled trials or high quality evidence. For example, there are no trials comparing checking blood glucose at intervals of 1 versus 2 h after increasing the insulin drip by 1 U/h when the blood glucose is 178 mg/dL and the prior blood glucose value 2 h ago was 138 mg/dL. Normally clinicians fill in the large gap of knowledge by extrapolating based on their knowledge of physiology, clinical outcomes, or other factors. The gap of knowledge, implicitly filled in by bedside clinicians, must be explicitly filled in during the eProtocol development process. Lack of definitive data to show a superior approach does not result in lack of strong beliefs regarding certain approaches by clinicians. Face-to-face meetings and consensus building are important. Participants must agree to develop reasonable rules when data regarding a superior approach are lacking. A reasonable, rather than a “perfect” or a “correct” protocol is the goal. During the iterative refinement and validation process of the eProtocol, these reasonable rules will be evaluated against the outcomes. Even though many of the individual, reasonable eProtocol rules are not based on evidence, the performance data for the intact eProtocol are ultimately more credible than the original data supporting the rules themselves.

Reasonable rules can be developed by consensus, building on credible published evidence, and expert opinion. Following Institutional Review Board (IRB) approval, and perhaps approval by a local software oversight committee, eProtocol can be tested in a suitable clinical setting. eProtocol can then be iteratively refined, and validated with broader clinical implementation. Initial eProtocol refinement can start with manual data entry and batch testing using real or simulated data. All protocol rules should be evaluated with particular attention to values at the margins, or thresholds, of the rules to verify that the rules work as intended. Manual data entry testing is necessary, in addition to the batch testing, to evaluate the user interface. Manual review of the clinician and eProtocol disagreements then generates further eProtocol refinement. The number of errors in each direction (clinician and eProtocol) should be monitored to allow a more quantitative assessment of the risk versus benefit of using eProtocol. Once eProtocol is validated, a batch test of validated data (patient states) is identified as a “gold standard” of known outcomes. This “gold standard” can serve as a validation reference data set for future changes in the eProtocol rules. As eProtocol is updated, the new version can be run against this “gold standard” output to ensure that unintended discrepancies do not occur.

Once eProtocol has been developed, further validation in clinical practice can be implemented with updated IRB and perhaps local software oversight committee approval. Prospective validation in a controlled clinical environment captures reasons for clinicians declining eProtocol recommendations. Manual review by knowledge engineers of the declined recommendations and further logic modification may be needed. Then, eProtocol can be exported to other clinical environments to broaden its use contexts, while continuing to monitor adverse events and outcomes. With each modification of eProtocol rules, revalidation against the batch test data “gold standard” and subsequent updating of the “gold standard” batch test data will ensure that only intended changes in the eProtocol are implemented.

Developed and validated eProtocols in use can be regularly updated and maintained (at least annually), perhaps as part of annual professional society activity, or by a formal group of experts with an eProtocol champion. After each update, eProtocol can be revalidated prior to clinical use.

Validated reproducible clinical methods will need to be stored in a reliable repository that can be accessed by clinician users. This repository might be hosted by organizations such as the US National Library of Medicine, the Agency for Healthcare Research and Quality (AHRQ), or even individual professional societies. This repository should provide around-the-clock support for web services that allow participating institutions to access the validated protocols.

7.4 Coordinate multiple eProtocols

As more eProtocols are developed and used clinically, we will need to coordinate use of multiple simultaneous eProtocols. We need to organize recommendations from different but simultaneously operating protocols, both in order of importance and in order of time, for action by bedside clinicians to fit with clinical workflows. Integration of multiple eProtocols will require construction of a formal hierarchy of clinically pertinent eProtocol recommendations. We believe this hierarchy will require construction of a formal set of eProtocol recommendation attributes that include risk (threat to the patient), benefit, urgency (temporal importance) and others.

7.5 Reform professional society guidelines

Current paper guidelines are often vague, require clinicians to fill in logic gaps, and thus assure variation in decision-making. We argue that reducing unnecessary variation and stabilizing the process with eProtocols, even in the absence of data to support one course of action over another, is an important step in clarifying clinical decisions. If society guidelines were replaced by eProtocols that were easily portable across multiple institutions, eProtocols might be widely adopted. Future studies comparing one course of action to another could answer questions about the best approach more efficiently than is possible in our current clinical research and care environment. eProtocols that enable clinicians and clinical researchers alike to easily implement the best current available evidence would likely be more valuable than the current guidelines that clinicians fail to follow consistently, across a broad range of clinical issues [5, 50–54].

8 Summary

The past 26 years have seen development, validation, and use of eProtocols. However, much more needs to be done before eProtocol use becomes routine in clinical practice. eProtocols are important in improving clinician compliance with best current evidence in patient care. Even in the absence of clear evidence, eProtocols are an important step in reducing unnecessary variation in clinical decision-making and clarifying best care.

To facilitate and expand eProtocol use, we need to implement and use information technology standards that allow integration of eProtocols with multiple EMRs, thereby avoiding double data entry. We need a system for coordination of multiple protocol recommendations with a formal set of eProtocol recommendation attributes. We need a well-defined process for the maintenance of eProtocols by professional societies and academicians. We need to establish repositories for updated and validated eProtocols. Finally, we need reasonable and consistent guidelines regarding the need for and type of regulatory oversight of electronic decision-support tools. Resolution of these issues should help ensure that eProtocols and other decision support tools that enable reproducible methods in clinical care and research remain consistent with the best current clinical care evidence.