Background

Checklists are a simple way to reduce errors in complex high-risk environments. Widely used in aeronautics for decades, Gawande et al. recently publicized the use of checklists in medicine [1, 2]. These observational studies demonstrated improvements in safety and outcomes when integrated into operating room routines, both in high-resource and low-resource countries [1, 3]. A simulation study by the same group further suggests that checklists also significantly improve surgical care during emergency situations where rapid and correct decision-making is crucial to ensure good patient outcomes [4].

Building upon these experiences and advances in informatics and human factor engineering, a novel electronic tool, the Checklist for Early Recognition and Treatment of Acute Illness and Injury (CERTAIN), is being developed by a large international collaboration with the overall objective of standardizing the approach world-wide to the evaluation and treatment of critically ill patients, in accordance with best-practice principles [5]. Similar to surgical checklists, CERTAIN may be particularly beneficial in low-resource settings with a scarcity of formally trained personnel [6].

During the evaluation of acutely decompensating patients, CERTAIN guides health-care providers through a structured approach, starting with a primary survey (ABCDE) followed by a secondary patient survey consisting of reason for admission, past medical history and the patient’s problem list. The latter is the gateway to the clinical decision support embedded into CERTAIN. In CERTAIN software, selection of a syndrome on the problem list leads to on-demand display of point-of-care key information in an index card format in the center of the screen with recommendations regarding further diagnostic and therapeutic steps (Fig. 1). Similarly, point-of-care key information is readily available or “just one click away” for selected procedures and medications. The process of creating and maintaining concise, accurate and up-to-date information in index card format (“cards”) for clinical decision support in CERTAIN is thus the cornerstone to CERTAIN’s success. Our experiences and lessons learned regarding this process will be the subject of the remainder of this paper (information about other aspects of the CERTAIN project including more analytical evaluations are available elsewhere [5, 7–9]).

Fig. 1
figure 1

Panel a shows CERTAIN’s main display facilitating a structured approach to acutely decompensating patients. Panel b shows the integrated on-demand clinical decision support for the syndrome card “shock” in the center of the screen

Full size image

Methods

A priori we postulated that an ideal content management system for CERTAIN should have the following characteristics:

Content

The content should cover a wide variety of clinically important topics, be easy to read, contain useful point of care information based on up-to-date evidence and be validated by expert reviewers. The content would initially be produced and deployed in English but translated into local languages as the project develops. The content should also be applicable in care environments which may have resource limitations and adaptable to local circumstances. The information provided should be supported by key references, web-links, and videos (e.g. demonstrating procedures) as appropriate. Given our objective to provide structured bedside decision support based on existing guidance and evidence, we deliberately decided against a process involving the development of new guidelines.

Infrastructure

The data management system that supports the clinical content must restrict access to authorized personnel and allow frequent, automated back-ups. Information should be stored centrally with the capability to instantly update all the different CERTAIN platforms (software, mobile application, PDF-booklet available for download and print). The data management system must further provide ease of access for authors and reviewers to facilitate development and validation of the content.

Results

We developed content for the software using the following steps. First, Bonneton et al. undertook an international survey of critical care professionals to identify the most common medical syndromes, medications, and procedures in acute care [7]. This survey also assessed the types of information considered to be most pertinent by bedside care providers (e.g. diagnostic tests vs epidemiologic data) [7]. Based on these survey results, we defined three different card categories – syndromes, medications, procedures – and created templates for each (for examples see Additional files 1, 2 and 3). A Mayo Clinic-based research physician, the “content lead editor”, organized the development and validation process for all cards (see Additional file 4: Figure S1 and Additional file 5: Figure S2).

Creation

Specific cards are drafted by content matter experts utilizing primary literature and published guidelines identified via multiple databases (e.g. Medline, EMBASE, National Guideline Clearing House, Cochrane Library, Up-To-Date). Both authors and reviewers are selected from the “expert panel”, a convenience sample of (currently) 81 physicians and 6 pharmacists from various backgrounds (e.g. critical care, anesthesia, emergency medicine), including many physicians involved in an ongoing study implementing CERTAIN into clinical practice [5].

Review and proofreading

Completed drafts of cards are then validated using an open peer-review process (Additional file 5: Figure S2): each new card is assigned to 2–3 reviewers chosen from the expert panel. Reviewers have the option to request reassignment to other cards based on their preference or expertise. Minor reviewer comments (e.g. grammar or spelling issues) are directly corrected by the content lead editor. Major comments are resolved by open discussion involving the original author, the other assigned reviewers, and the content lead editor who is supported by, and can seek further advice from, the “content management panel” at any time. This latter group consists of five senior physicians who are also part of the expert panel. Although all major comments to date have been resolved using this process, for content issues which remain unresolved, a mechanism exists to facilitate final arbitration via a modified Delphi process involving the entire expert panel [10].

Publication and updates

After a card is proofed and finalized, it serves as the blueprint to update the different CERTAIN platforms. To ensure that the content stays up-to-date, authors receive a request after one year to update their card with regards to changes in current evidence and guidelines, followed by the same review process as when creating a new card. In addition, this updating process can be triggered at any time by any CERTAIN user via an embedded feedback button in the software. While the main work flow for development and validation is based on the English content, cards are currently being translated simultaneously to other languages including Spanish, Chinese, Turkish, Croatian, Serbian, and Polish. We currently have 60 Spanish, 176 Chinese,152 Turkish,20 Serbo-Croatian and 65 Polish cards. The process is similar to the general validation process: experienced bilingual clinicians translate the cards which are then reviewed by bilingual peers prior to incorporation into the various CERTAIN platforms (for more details see Additional file 6).

Ownership

The coordinating center overseeing content management across all platforms is the Mayo Clinic, Rochester. This role includes development, review, updates, and translation of content and infrastructure and involves system organization, reminding authors and reviewers about update deadlines, recruiting authors, reviewers and translators for new content development and working with programmers to find technical solutions for data management issues. Individual card “ownership” and authorship is shared between the author and reviewers and acknowledged on all CERTAIN output (software, mobile app, PDF). If any original authors/reviewers cannot maintain “ownership” of a card the coordinating center assigns new individuals to adopt that card.

Discussion

Card authorship

While the general process outlined above has remained constant over time, many practical details were refined due to changing circumstances and needs as summarized in Table 1. For example, one major change was the increase in the number of authors (See-Appendix 6 list of authors/reviewers). Initially, these were restricted to a few members of our research group to facilitate rapid growth and consistent standards of the card inventory. However, given the need for annual revisions, it has not been feasible for one author to take ownership of more than just a few cards. The current number of cards created by 36 authors includes 67 syndromes, 117 medications, and 30 procedures (for a full list of cards see Appendix 5). Thus, over time we recruited more collaborators, often colleagues of original authors and reviewers, and trainees at the fellow level in the pulmonary and critical care department at our institution. So far 127 (59 %) cards have been reviewed, with each reviewer being assigned an average of two cards (range 1–7 cards).

Table 1 Summary of the evolution of the content management system
Full size table

Software infrastructure

Cards were initially created, stored and modified in Microsoft Word® using Dropbox® links for sharing. Using those links, reviewers could download the Word® files, revise and add comments to the documents and then send them back to the content lead editor as an email attachment. Unfortunately Dropbox® did not support real time collaboration between different authors simultaneously. While technologically simple, this method became impractical due to the large amount of time needed to organize the collaboration process to enable reviewers to see each others’ comments and hence facilitate a virtual discussion. The potential for missed emails and slow file updates, and the lack of version control became unworkable for the content lead editor. We improved this process by switching to Google Docs®, which allows sharing of text documents via links and editing of the source files directly by multiple reviewers simultaneously. Drawbacks of this approach included loss of much of the initial formatting and the inability to easily create and update a PDF booklet. Microsoft Word® allowed linkage of all single documents together in one master file with an automated table of contents, creating a booklet which could be updated within minutes. In addition, although freely available in general, access to Google Docs® is restricted in certain countries (e.g. China) where some of our authors and reviewers live, thus limiting some collaborative opportunities.

Although not directly applicable to our project, because the various open source content management systems reviewed by Mooney et al. are geared towards creators of wikis or blogs, we agree with their final conclusion that when choosing an appropriate tool “first and foremost, security should be evaluated as well as the aptitude, availability and coverage of the user support community” [11]. Neither Google docs® nor Dropbox® met our security needs, to which there are several aspects: We want to be certain that cards can only be edited by invited and qualified individuals to ensure high quality of the cards which potentially impact clinical decision making and thus health outcomes; additionally we also want to enable local clusters (e.g. a specific hospital) or users to customize the content to reflect local/individual preferences with those modifications being editable and visible only by that local cluster or user, respectively; lastly the data needs to be backed up on local servers automatically at frequent intervals to ensure restorability and offline work in case of disrupted internet connection and/or failure of the cloud server etc. Of note, with regards to our content management system we are not concerned about any patient confidentiality issues, because while personal data can be entered into CERTAIN software (to facilitate charting and debriefing) the cards themselves do not contain any patient information.

Furthermore both Google docs® and Dropbox® required excessive work to execute simple tasks, including manually updating the software and mobile applications whenever content changed; sending email reminders to reviewers to ensure compliance with peer review deadlines; and keeping track of the cards’ expiration dates. Due to the absence of readily available software meeting our requirements, we have thus started developing a customized software solution that will allow authors and reviewers to co-create, edit and review content directly in a secure cloud server with the capability to automatically update the different platforms including the PDF booklet (see Table 2 and Additional file 7: Figure S3).

Table 2 Infrastructure of the customized content management system
Full size table

While we have been somewhat struggling to find the right infrastructure, we have made much progress in the development and validation of the content itself, despite essentially no funding. Within three years we were able to create more than 200 cards, with more than half of them being validated. In large part this was only possible by directly involving many of the CERTAIN end-users into the content-creation process, thus reducing the work load per person while increasing the users’ “buy-in” into the CERTAIN concept.

While our approach of an open peer-review may increase the risk of “herding” (ie. reviewers being more likely to be influenced by and agreeing with their peers’ potentially incorrect opinions), [12] we intentionally adopted this process because open peer-review is generally thought to increase accountability, fairness, and transparency, with some evidence showing that it leads to better quality reviews, while being preferred by authors [13–16]. Despite the theoretical risk that identifiable reviewers may feel more hesitant to criticize their peers, in practice the loss of anonymity does not appear to significantly affect reviewers’ decision to ask for major revisions or reject manuscripts [15]. While “open peer review” generally just denotes that reviewers’ identity is revealed to the authors (and possibly to the readers if the manuscript under review is eventually accepted) we further enabled reviewers to be aware of each other’s identity and opinions as well. This may further increase the risk of “herding”, but at a time where medical knowledge doubles approximately every 3 to 4 years we feel it is best to have an open and maximally transparent discourse involving as many content experts as possible [17]. This principle of maximal inclusion of content experts also underlies the mechanism to resolve complex issues using a modified Delphi process involving the entire expert panel and our efforts to encourage each CERTAIN user to simultaneously function as a peer-reviewer via the feedback option within the software.

A crucial challenge in developing globally applicable decision support is to provide best-practice recommendations that are relevant in low-resource practice settings where some tests and interventions may not be available. We tried to solve this conundrum by crafting cards based on best evidence assuming a resource-rich setting while allowing users to add permanent contextualizing card notes to each card. In the future we further plan to add this feature for clusters, so that for instance hospitals can add a specific recommendation regarding antibiotics for pneumonia taking into account local resistance patterns and drug availability, which will be visible to all providers affiliated with that particular hospital or cluster.

There have been many other attempts to harness the technologic progress to improve and standardize health-care in under-developed and under-served regions: for example, within the United States about 6 % of all intensive care patients are now (co-)managed by a telemedicine intensivist who remotely monitors patients’ condition in real-time and provides support to the onsite personnel [18]. This remote expertise and standardization of care appears to improve outcomes, but major barriers include the need and cost for 24/7 available remote experts, variable acceptance by the onsite personnel (who may feel monitored rather than supported) as well as variable integration and interoperability with the local software environment [18]. In an interesting extension of the telemedicine concept to resource-poor countries Celi et al. realized that a major infrastructure asset is the rampant availability of mobile phones which allow access to and interchange of information even in settings devoid of wireless internet and computers, and thus created a “cell phone-facilitated clinical system” [19]. This system, which is integrated into open MRS (an open source medical records system), allows users (e.g. patients or local health workers) to send medical information including images and voice messages to remote specialists who in turn could provide live decision support. While we similarly encountered software issues described above, CERTAIN alleviates the acceptance issue by providing on-demand, interactive best practice advise to the onsite providers, empowering them to use, ignore or modify the information at their own discretion. Additionally, and similar to the project by Celi et al., CERTAIN tries to address the challenge of getting technologic support to areas with potentially minimal infrastructure and/or internet capability by offering decision support via various platforms including a paper version as well as a mobile phone app.

Another major challenge at the intersection between technology and healthcare that we encountered with CERTAIN itself is the difficulty of keeping a balance between doing justice to the complex environment that it is being designed for (i.e. evaluation of critically ill patients) while keeping the interface simple since in these situations time is generally of the essence. For example, during a recent simulation study participants largely provided positive feedback about CERTAIN in general, but felt that the software should become somewhat more intuitive. It is reassuring though, that in this simulation study CERTAIN improved health-care providers’ performance [20]. It is unclear, however, whether the benefit is due to its embedded decision support or the other components such as teaching a structured approach to patient care, safety culture, and closed loop communication strategies. A before-after quality improvement study is underway to assess the impact of CERTAIN on care processes and patient outcomes when implemented into clinical practice in multiple Intensive Care Units (ICUs) across five continents after training local personnel remotely via live video stream [5, 9]. This study will significantly increase the number of users who can provide feedback about how well the decision support aligns with frontline providers’ needs. Based on this feedback, we are continuously improving the decision support system with a special focus on workflow integration, data entry and output, standards and transferability, and knowledge maintenance [21]. However, if shown to improve processes of care or patients’ outcomes, future research will be needed to determine the relative contribution of CERTAIN’s different components.

Conclusions

Although we were able to successfully develop and validate a large inventory of clinical decision support cards in a short period of time, readily available software products are suboptimal for use as content management platforms, requiring us to pursue a customized software solution.