Introduction

Registration of surgical complications is important to assess and improve quality of surgical care [1]. Also, analyzing surgical complication registration outcomes can and should lead to improved patient outcomes [2]. Despite widespread acknowledgement that complications should be reduced, controversy exists how to detect and record these complications [3].

Traditional efforts to detect complications have focused on voluntary reporting or incident reports [4]. These methods have often been poorly successful in the detection of complications [3]. For example, registration during verbal hand-off meetings yields a registration rate of only 86 % of all complications [5]. Moreover, to achieve adequate reporting of complications, a sufficient number and diversity of surgeons should participate in the daily verbal hand-off meetings, but this is time-consuming for highly qualified surgeons. Hence, hospitals would benefit from a more effective way to identify complications and to complete their registration.

An attempt to design a more uniform, practical, and efficient complication registration method came from the Institute for Healthcare Improvement (IHI), who developed the Global Trigger Tool (GTT) [4]. A “trigger” can be defined as a specific factor that is derived from the patient’s medical record and is associated with an increased risk for complications. These factors can be patient-specific (e.g., lab results, BMI), surgical procedure-specific (e.g., complexity of the procedure), or hospitalization-specific (e.g., length of hospital stay). A “trigger tool” is a set of triggers that identifies patients who are likely to have suffered a complication and thereby indicates which patient records should be checked for complications, for instance by a data manager.

The benefits of (some form of) the GTT to detect complications have been studied in terms of inter-rater reliability among different reviewers on reporting complications. [3, 6–12]. Two studies showed a high specificity (92.0 and 99.0 %), but low sensitivity (23.0 and 28.0 %) [13, 14]. The high specificity means that the methods could be used to replace expensive manual chart reviews because less “falsely positive” charts need to be checked. However, in order not to miss any complications, the sensitivity of the method should also be high.

The aim of this study was to develop a new trigger tool to assess the accuracy and usefulness compared with the verbal inventory.

Methods

Patients and setting

This study comprised a model development and diagnostic accuracy study, based on a 1-year sample of hospitalized surgical patients. The study was performed at the department of surgery of a tertiary referral university hospital in Amsterdam. All patients (n = 4534) above the age of 17 admitted to or operated by a surgeon from this department between July 2012 and June 2013 were included in this study. This surgical department provides general, gastrointestinal, hepatopancreatobiliary, vascular, and trauma surgical services.

Verbal inventory and registry of complications

Currently, surgical residents collect preoperative, intraoperative, and postoperative data for each surgical patient real-time. The attending staff may supplement, during the morning hand-off, the complications identified by the residents. Subsequently, the database manager reviews, in retrospect, the charts of the patients identified with a complication for possible additional complications. The data manager uses definitions and agreements about specific clinical situations. In case of uncertainties, the data manager consults the surgeon responsible for the registration. The identification and collection of complications by surgical residents and during the morning hand-off is defined as “verbal inventory.”

All complications are registered and categorized by severity based on the Clavien-Dindo classification in the departments’ complication database [15]. The complication registry categorizes each complication into four grades of severity: grade 1, temporary health disadvantage recovering without reoperation (grade 1 management includes radiological or endoscopic interventions; similar to Dindo grade I, II, and IIIa); grade 2, recovery after reoperation (similar to Dindo grade IIIb); grade 3, (probably) permanent damage or function loss (similar to Dindo grade IV when permanent); and grade 4, death (similar to Dindo grade V).

A “complication” is defined according to national and international standards as “an unintended and unwanted outcome or state during medical care that is so harmful to the patients’ health that it requires (adjustment of) treatment or leads to permanent damage” [16]. Complications that occur after discharge are not registered unless the patient is readmitted within 30 days after discharge.

Development of the trigger tool

A set of potentially relevant triggers was chosen based on (a) a previous systematic review of the literature in which these triggers were found to be significantly associated with surgical complications [17], (b) questionnaires containing the potential triggers found in the literature answered by 12 surgeons within our hospital to validate or supplement this set of potential triggers, and (c) availability of the trigger in electronic hospital databases. Correctness of the department’s database on complications was monitored by two investigators. Subsequently, two investigators independently extracted the values of the set of potential triggers from hospital databases belonging to the records of patients in the study period. To ensure correctness of the data entry, one investigator (AES) checked a random set of 20 patient records per trigger as entered by the other investigator (CMvL) and vice versa. Univariable logistic regression analysis was performed to find triggers that occurred substantially more often in the group with complications in the departments’ complication database. Variables with p < 0.20 were entered into a stepwise multivariable logistic regression analysis to find significant independent triggers (p < 0.05). This association was expressed as their odds ratio (OR), 95 % confidence intervals, and p values. Continuous variables were dichotomized. Different ranges of cutoff values (based on clinical relevance and literature) were tested for each variable and analyzed with different cutoff limits. The most significant value was set.

Reference standard

Of all patients with complications, 86 % were recorded in the current database [5]. In order to validate and compare the trigger tool, a reference standard was formed by a random sample of 150 records with complications and 150 records without complications, meaning oversampling for complications. For each potential trigger, a minimum of 10 records (as a rule of thumb) should be included in the reference standard in a regression analysis. A sample of 150 records was considered to be adequate as a reference standard to be able to detect 15 possible independent predictors.

The medical files were reviewed by two investigators (AES and CMvL). If the investigators did find a complication in the latter 150 records, this record was discarded and added to the records with complications. In case of uncertainties interpreting the texts of the resources about complications, the investigators consulted each other or their supervisors. This procedure was continued until the group without complications also contained 150 verified records.

Validation of the trigger tool

The independent triggers from the departments’ complication database were subsequently entered in another multivariable analysis, now using the reference standard in order to check their validity. Triggers were kept in the model if they again contributed significantly to the model. If not, we decided to remove the trigger from the model unless the trigger had a low incidence (<10 patients) in the reference standard and was a significant factor in the univariable analysis. The remaining independent triggers formed the final trigger tool.

Comparing the verbal inventory with the trigger tool

The verbal inventory and trigger tool were compared with the reference standard to calculate their sensitivity and specificity as to the detection of one or more complications and the severity, type, and number of complications registered.

All statistical analyses were performed using IBM SPSS Statistics v.20 (IBM, Armonk, NY, USA).

Results

Patients and setting

A total of 4534 patients admitted to the hospital between June 2012 and July 2013 were included in this study. Their mean age was 55 years (range 18–99). Of these, 2529 (55.8 %) were men and 2520 patients (55.6 %) underwent operative treatment. In 795 of the 4534 (17.5 %) patient records, one or more complications were documented in the departments’ database.

Development of the trigger tool

The systematic review provided 25 potential triggers that were significantly associated with the occurrence of surgical complications [17]. The inventory among the hospital’s surgeons yielded nine additional potential triggers (Fig. 1). This led to a total of 34 potential triggers for data collection. Of these, 21 were readily available in hospital databases and extra 10 specifications of the potential triggers were available (Appendix). For example the trigger emergency operation resulted in two triggers: (1) emergency operation during admission and (2) the highest urgency classification of emergency surgery during admission.

Fig. 1
figure 1

Flow chart of the triggers eventually included in the trigger tool

Full size image

Univariable analysis found 23 out of 31 potential triggers with a p <0.20 (Table 1).

Table 1 Outcome of univariable analysis of variables possibly associated with complications, expressed as p values and 95 % confidence intervals (CI). Study group N = 4534
Full size table

Some of the triggers influenced each other, for example, an esophageal resection was associated with admission at the ICU and a higher complexity of surgery. To study their separate effect, we had to develop three models: one general model for patients who underwent a surgical procedure, a second focusing on specific surgical procedures that were prone to result in postoperative complications, and a third for patients who did not undergo surgery, but who could still be burdened with the occurrence of complications during the admission period or in 30 days after discharge (Table 2).

Table 2 Multivariable analysis using the departments’ database. Study group N = 4534
Full size table

This resulted in 11 independent triggers, 4 of which were continuous variables: length of hospital stay, extension of standard surgical procedure time, complexity of procedure, and age. Their cutoff values were set to the following: length of stay ≥14 days, technical complexity of procedure ≥class 6, age ≥85 years, and time required above the scheduled procedure time ≥110 min. Table 2 shows the results of the trigger tool with the 11 potential triggers again tested in the multivariable analysis, now dichotomized based on these cutoff values.

Validation of the trigger tool

The triggers were tested using the reference standard (Table 3). The reference standard is not representative for the total population in the departments’ complication database due to oversampling of records with complications. Due to the smaller size of this group, we also studied the incidence of the triggers. The multivariable analysis, now using the reference standard, showed 6 out of the 11 potential triggers to be significant (Table 3). The triggers “time required above the scheduled procedure time ≥110 minutes” and “age ≥85” were found not to be significantly associated with the presence of complications (p > 0.20). The incidence of 3 potential triggers was low, but these triggers were significant in the univariable analysis. Therefore, these triggers were nevertheless included in the trigger tool (i.e., esophagectomy (n = 6), Whipple procedure (i.e., pancreatoduodenectomy; n = 7), and abdominal aortic aneurysm surgical procedure (n = 6)).

Table 3 Multivariable analysis using reference standard. Study group N = 300
Full size table

Thus, nine significant independent triggers were included in the final trigger tool, containing as follows: emergency procedure, complexity of surgical procedure above class 6, do not resuscitate policy (DNR), ICU-stay, length of hospital stay of more than 14 days, reoperation, esophagectomy, Whipple procedure, and acute (or ruptured) abdominal aortic aneurysm surgical procedure.

Receiver operating characteristic (ROC) curve analysis was subsequently performed to determine the number of positive triggers needed to detect complications most accurately. The trigger tool performed best already if one out of the nine triggers in the trigger tool would be present (sensitivity 70.7 %, specificity 70.0 %; AUC 0.764, Fig. 2).

Fig. 2
figure 2

ROC curve of the number of positive triggers needed to detect complications

Full size image

Comparing verbal inventory with trigger tool

Patient records

The sensitivity values of the verbal inventory and trigger tool methods to detect complications as compared to the reference standard were 78.7 % (118/150) and 70.7 % (106/150), respectively, while specificity values were 100 % (150/150) and 70.0 % (105/150), respectively. Hence, the verbal method would miss 21.3 % of the records with complications, while the trigger method would miss 29.3 % (Table 3).

The sensitivity to detect records with major complications (severity class ≥ 2; reoperation,) was higher for the trigger tool than for the SCR, 97.2 and 80.6 %, respectively (Table 4).

Table 4 Verbal inventory versus trigger tool, sensitivity. Study group N = 300
Full size table

If a combination of the trigger tool and the verbal inventory was used, 138 out of the 150 records with complications would be detected (sensitivity 92.0 %).

Missed complications

The verbal inventory missed 31 records with one or more complications (in total 71 complications). The trigger tool missed 45 records (in total 53 complications). All complications missed by the trigger tool were minor complications (severity class < 2), especially wound problems. The verbal inventory also missed mainly minor complications, but these were categorized as functional disturbance (i.e., hypertension or electrolyte derailment). Two severe complications were also missed by the verbal inventory (reoperation and death).

Discussion

Based on our study results, the proposed trigger tool appears as accurate as a verbal inventory in terms of sensitivity and specificity as to the detection of complications that occur in hospitalized surgical patients. Furthermore, the trigger tool we developed detected a higher number and a higher proportion of more severe complications. Only some mild complications would have been missed, for example, wound infection with no need for a reoperation or cardiac complications.

On the other hand, the verbal inventory during the morning hand-off provides awareness of the complications suffered by their patients and deliberation and reflection with and by their colleagues. The two methods were tested with regard to their ability to identify records with complication(s). To optimize the registration of complications, the results must be referred to the surgeons, who should discuss these in order to undertake preventive actions in, for example, separate complication meetings.

This study is one of the few that used a reference standard to assess the comprehensiveness of the detection of patients with complications by either method. Most studies have used a “silver” standard or even no reference standard at all [13, 14, 18]. The sensitivity of the trigger tool in this study compares favorably to other forms of term searching tools, such as scanning the discharge letters for words suggestive for complications [13, 18], or a natural language processing detecting method [14].

As an alternative to the detection methods investigated here, the clinical observation method is the investigation of potential complications by a trained observer of all patients and providers, who is alerted by a predefined list of clinical event “triggers.” Clinical observation is a powerful tool for identifying incidents and errors in medical care, especially when compared with self-report or voluntary reporting mechanisms [14, 19]. This observation method, however, uses clinical observers and also focus groups to identify complications, which would imply a huge manual effort. Another option, the IHI collaborative “Global Trigger Tool” (GTT), appears to detect more complications than other conventional approaches but requires substantial manual effort [3, 6, 7].

The proposed trigger tool (tailored to a Dutch hospital context) is expected to be highly resource-intensive and will require manual database searching by the database manager whereas the surgeons’ workload will probably minimize. Further research is needed to determine the time and costs. A fully electronic database including all admitted patients and their health care utilization characteristics would facilitate the use of this trigger tool. Unfortunately, symptoms, diagnoses, and physical findings are usually recorded as narrative texts, but are yet unavailable in coded form. Nursing files and surgical discharge letters were found very helpful to find complications [5]. The trigger tool method could be simplified by the retrieval of complications from patient nursing files and surgical discharge letters. In addition, nurses might play an important role in the process of complication registration. Application of the trigger tool should also be simplified by systematized electronic storage information in hospital data systems, in order to detect patients “at risk” more easily.

Limitations

The study is a single-center study. However, the predictive factors used as potential triggers were derived from a systematic review based on studies from various institutions. Even if not every trigger (for example, highly specialized surgical interventions performed only in high-volume centers) is relevant in some centers, the remaining ones may well be. Three of the triggers equal three major surgical procedures in a university hospital. These procedures are not undertaken in all hospitals. But there has been a tendency for health systems to rush towards inadequately justified “one fits all” solutions [13], but the effectiveness of these global tools vary widely and does not provide the robust data hospitals need to base their decisions on to optimize patient safety. However, the absence of certain triggers or surgical procedures in some hospitals does not imply this trigger tool is invalid. The use of a customized set of triggers to detect surgical complications, as provided in this study, could result in high sensitivity, especially for the detection of major complications. The potential triggers found in a previous systematic review can be used as a set of potential triggers [17], and this article provides the method for other hospitals to develop a customized trigger tool based on their own type of surgery.

Furthermore, the definition of the DNR status may be culture-sensitive. We do not, however, think this will have a major impact on the validity of the trigger tool, but only on the predictive power of this trigger only.

Although tested on a reliable reference standard (n = 300), little is known about the risks of the trigger tool in a larger population regarding missed complications. For practical reasons, the number of patients in the reference standard was limited, which may have led to fewer significant triggers. Further external validation is warranted to assess the value of this trigger tool before it can be implemented in clinical practice. The reference standard datasets are a subgroup of departments’ complication database. This procedure might not be statistically appropriate but incorporates only 3 % of the total patient population. Therefore, we assume the influence will be small.

Cutoff values were based on practical grounds and clinical relevance. Determining cutoff values by means of ROC analysis would have resulted in identifying more than 70 % of patient (records) that might have suffered a complication (e.g., the optimum cutoff value for age would be >57, while the patients’ mean age was 55).

This trigger tool is very helpful after discharge in the detection of complications by identifying high-risk patients’ records. Another use of “triggers” could be their functioning as so-called red flags, highlighting the patients who are sensitive for developing complications, which could be useful in the improvement of complication prevention in clinics. Not all of our triggers could function as red flags before admission. The type of surgery and planned IC admission are known before admission. Nevertheless, if these red flags occur during admission, the patient can be marked as “high risk” for developing complications, either to detect them early or even to take actions to prevent (more) complications during admission. Although studies on the GTT are common, little is known about the use of a customized trigger tool [16] or sensitivity outcomes against a reference standard. The need for an efficient and inexpensive means to detect complications makes further research on an electronic (trigger tool) approach attractive.

Conclusion

The use of a customized set of triggers as proposed here for a university hospital to detect surgical complications results in high sensitivity for the detection of major complications in an academic hospital. On the other hand, mild complications would be missed, for example, wound infection with no need for a reoperation. The proposed trigger tool appears as accurate as a verbal inventory in terms of sensitivity and specificity as to the detection of minor and major complications. This study provides the method for other hospitals to develop a customized trigger tool based on their own type of surgery to detect more severe complications, provided that it is simplified by a systematized electronic storage of patient characteristics and trigger valuable information in hospital data systems.