Anesthetic records are vital for the clinical management of patients both in theatre and postoperatively, and they provide an important source of information for audit, research, administrative, and medico-legal purposes. Fifteen percent of anesthesiologists intraoperative time is dedicated to compiling handwritten records.1-3 Automation of this task can reduce the time devoted to completing anesthetic records4-6 while increasing the quality of data contained in them.7-10 Anesthesia information management system (AIMS) technology was once the domain of an enthusiastic minority, but use is increasing.11,12 Published data have reduced concerns about the potential of AIMS to reduce vigilance and increase medico-legal exposure5,13,14 (although there were more lapses in response to a vigilance task in a recent randomized controlled trial [RCT] in our own institution).6 Sophisticated AIMS have integrated initiatives to increase adherence to guidelines, reduce errors in drug administration, and facilitate quality assurance.6,15-20 Since mid-2005, our own institution has used SAFERsleep® (Safer Sleep LLC, Nashville, TN, USA), a locally developed multimodal AIMS, for most anesthetics; this system has been described in detail elsewhere.6,21

Several studies have compared the accuracy and completeness of physiologic data in handwritten and AIMS records and have found AIMS records to be superior.7-10 This is not surprising, as electronic data from physiologic monitors lend themselves to automated capture by an AIMS. It is less clear whether AIMS technology facilitates recording information that can be obtained only by clinical observation. Entries of this type are user-generated, usually as text entries within the record; however, for some practitioners, writing may be more reliable than typing or choosing items from drop-down menus. Pen and paper are easy to use for this type of information, and many manual anesthetic records (including the one formerly in use in our institution) have been carefully designed to facilitate the complete collection of important items of information.22,23 In the AIMS used in this study, data are collected using a combination of bar code scanning, drop-down menus, and free text entry. It is possible that computer technology might make this particular part of recordkeeping more prone to certain types of error, for example, inadvertently accepting default options that may not apply to particular cases or assuming that information has been included in automated information templates when in fact it has not.

The abovementioned RCT6 provided an opportunity to investigate the difference between the completeness of specified information in text entries in AIMS with handwritten records made by the same practitioners. We also evaluated the combined effect of direct observation and other research procedures (often cited as a potential confounder in clinical research) on the completeness of this information.

Methods

Trial design and participants

This was a substudy of an RCT that was described elsewhere in detail6 and conducted with approval from the Northern Y Regional Ethics Committee (NTY/07/10/112, November 2007). The outcomes of this substudy were included in those listed for the trial registration of the overall RCT.

As mentioned above, the AIMS used for this study is the SAFERsleep system. The handwritten records were compiled on blank locally designed anesthetic record sheets used at the study institution from 2003 until the adoption of the AIMS and still available within the institution for use if needed. Records were considered complete following sign-off by the anesthesiologists in the postanesthetic care unit, at which point they are legally held in the study institution to represent the finalized clinical record of the intraoperative period.

Key aspects of the methods have been described previously.6 Records were compiled during cases in designated operating theatres in the adult anesthetic department of Auckland City Hospital. Operating theatres included in the study were selected to give a broad case mix, and the study statistician used a computer-generated random number sequence to randomize the operating theatres by week for use either with the AIMS or with handwritten records. All participating anesthesiologists provided informed consent and received formal training in the use of the AIMS.

From the 1,075 records completed for the parent RCT, a sample of 200 cases from each study arm was included in the current analysis (total n = 400). The first 100 and last 100 consecutive non-cardiac general anesthetic cases in each arm were selected to reflect changing levels of direct observation during the parent study. Observers were present to collect workload and task analysis data throughout the first 416 cases included in the main RCT. This involved close observation of anesthesiologists and entry of observations into a highly visible tablet computer. Study personnel were present for most but not necessarily all of the remaining 659 cases and did not collect workload data or enter observations into these computers. Consequently, their presence and observational activities were substantially more overt during the first 200 cases evaluated (n = 100 from each arm) in this substudy than in the last 200.

Records were accessed from the electronic medical record system where they are stored as scanned copies of the original clinical record (i.e., records were evaluated as they are retained for clinical and medico-legal purposes).

Analysis

Our primary outcome measure in this substudy was the number of items that complied with a checklist of 32 items used to evaluate the record. The checklist was an adaption of the PS06 document (2001), “Recommendations on the Recording of an Episode of Anaesthesia Care”, that was produced as a guideline by the Australian and New Zealand College of Anaesthetists (ANZCA)Footnote 1 and outlines the information that should be recorded on the anesthetic record.

The study checklist was developed and refined in a pilot comparison of records created before the introduction of the AIMS with those created soon after its introduction. Information that did not pertain to every procedure was excluded from the analysis, as were items identified as usually recorded pre- and postoperatively.

The anesthetic records were evaluated by three of the researchers (K.E., S.H., J.H.) who had some familiarity with anesthesia. They were required to read an introductory textbook on anesthesia and observed at least 20 cases in the main RCT before beginning data analysis. Data identifying clinical staff or the patient were marked as present or absent, and then the applicable records were de-identified. Remaining items on the checklist were assessed for completeness and graded one if complete, zero if absent, and 0.5 if incompletely recorded. Items recorded in the pre-anesthetic assessment sheet but not in the intraoperative anesthetic record were considered to be complete. The “description of airway” item was marked as not applicable where a laryngeal mask airway device was used. The “name of supervisor” and “level of supervision” items were marked as not applicable in cases where a specialist anesthesiologist was the only clinician providing anesthetic care. The primary outcome measure was reported as a percentage completion of relevant checklist items to reflect the changing denominator with the inclusion or exclusion of these items. Only those items for which reasonable certainty was possible were scored, and if any items were in doubt, the record was evaluated with the assistance of a specialist anesthesiologist (C.K.).

Records were also evaluated for 1) the total number of intravenous drugs listed; 2) the number of intravenous drugs for which a dose could be determined; 3) the total number of bolus administrations recorded; and 4) the number of intravenous boluses for which administration time was determinable.

Ten records from each group were randomly selected and reassessed by a specialist anesthesiologist (C.K.). These data were compared with the other researchers’ assessments (K.E., S.H., J.H.) to verify data accuracy and inter-rater reliability of the checklist.

The distribution of data was examined and summary statistics were calculated (SPSS Inc., Chicago, IL, USA). The primary outcome measure of the two recordkeeping methods was compared using a general linear model comprising recordkeeping method, presence of observer, and participating anesthesiologist as factors. Secondary outcome measures were compared using Student’s two sample t tests for continuous data. Discrete data were analyzed as indicated in the text using Chi square or Fisher’s exact tests (where one or more expected cell frequencies were < 5). As multiple comparisons were performed, the criterion for statistical significance for secondary outcome variables was adjusted using a Bonferroni correction.

On the basis of data collected during the pilot comparison, an overall within-group standard deviation of 12% was assumed for percentage compliance with the checklist. For the primary outcome comparison, 92 participants per group would give 80% power to show a difference of 5% in overall compliance score with a type I error rate = 0.05 (two-sided). This was rounded up to 100 participants per group.

Results

Data were collected from cases managed from March 2008 to February 2009. Patient characteristics, case difficulty, and case length were reasonably well balanced between the two groups (Table 1). Anesthesiologist experience and case-mix were balanced between the two arms and between the two sampling periods (i.e., “Intensive Observation” and “Reduced Intensity Observation”). The 400 records were produced by 69 participating anesthesiologists: 56 anesthesiologists contributed 1-9 records each; nine anesthesiologists contributed 10-19 records each; two anesthesiologists contributed 20-29 records each; and two anesthesiologists contributed 30-39 records each.

Table 1 Baseline characteristics of patients by recordkeeping method
Full size table

There was good concordance between data entered by the researchers (K.E., S.H., J.H.) and a specialist anesthesiologist (C.K.), and only one discrepancy was identified, which related to the total intravenous drug count in a handwritten record.

Primary outcome variable

The mean percentage compliance score was 88.6% in the AIMS records and 81.5% in the handwritten records (mean difference 7.1%; 95% confidence interval [CI] 5.6 to 8.6%; P < 0.0001). Intensity of research observation and participating anesthesiologist were highly significant determinants of mean percentage compliance score in the general linear model (P < 0.0001). All of the checklist items were recorded in 3.5% of AIMS records and no handwritten records. Several handwritten records required reference to the pre-anesthetic assessment, whereas all information was contained in the intraoperative AIMS record. Six items were recorded in 100% of records: name of hospital, hospital record number, intravenous drugs listed, use of pulse oximetry, and monitoring of blood pressure and heart rate. Following Bonferroni correction, completeness of recording between groups was significantly different (P < 0.0015) for six individual items (Table 2). The remaining items were recorded in fewer than 100% of records without statistically significant differences in level of completeness between groups (Table 2).

Table 2 Percentage completion of the checklist overall and by individual items by recordkeeping method
Full size table

Secondary outcomes

There were no clinically important or statistically significant differences between groups in intravenous drug administration endpoints (Table 3).

Table 3 Intravenous drug administration data (mean number of entries per record) by recordkeeping method
Full size table

The mean percentage completion of text entries was significantly greater for records produced during the period of intensive intraoperative observation (87.3%) than for records produced during the subsequent period of less intensive observation (81.6%) (mean difference 5.7%; 95% CI 4.2 to 7.3%; P < 0.0001). This difference in quality of recordkeeping was present in both recordkeeping methods but was more pronounced in the handwritten records – in fact, completeness was comparable between the handwritten records produced with an observer present and the AIMS records produced without an observer present (84.8% vs 85.9%, respectively; mean difference 1.2%; 95% CI −2.8 to 0.5; P = 0.16) (Figure).

Figure
figure 1

Mean percentage checklist completion by recordkeeping method and observation cohort

Full size image

Discussion

In this study, when the data were compared with an accepted clinical guideline, the manually entered data in the AIMS records were more complete than the data entered in the handwritten records. Nevertheless, there was room for improvement in the overall completeness of records regardless of recordkeeping method. For example, estimated surgical blood loss was recorded in < 20% of both AIMS and handwritten records. Records produced early in the study during a period of more intensive research observation were more complete than those produced in the final stage of the study (with less intensive observation); the causes of this effect are probably multifactorial. Unsurprisingly, the participating anesthesiologist was a significant determinant of the completeness of the anesthetic record; however, the primary outcome measure remained highly significant after controlling for individual anesthesiologists.

Previous investigators have described deficiencies in the quality of handwritten anesthetic records in comparison with those produced by an AIMS.7-10 These studies predominantly relate to the accuracy of physiological data. In a recent study, completion rates were assessed for six text entry fields on AIMS records,24 and results showed rates similar to those seen in this study; however, randomized evaluations directly comparing the completeness of text entry data between different methods of recordkeeping are lacking. Results of a previous assessment of the AIMS used in this study using a smaller sample size (n = 120) showed no overall difference in completion between handwritten and AIMS records.25 That study evaluated AIMS records created following recent implementation of the AIMS system, and records were therefore produced by relatively inexperienced users of the system. There is a known learning curve associated with correct use of the system,21 and the inclusion of new users may explain the lower observed completion rates for AIMS records compared with those seen with experienced users in the current study.

There is potential for bias in the choice of items included in the checklist, and for this reason, our checklist was based on an authoritative guideline for anesthesia recordkeeping.A This increases the clinical relevance of the primary outcome measure; a 100% completion score would arguably be a reasonable target for high-quality anesthetic records and ought to be achievable. A limitation of the study is that no attempt was made to quantify the accuracy of the text entry data. The accuracy of text entry data on anesthetic records has been assessed previously (also in a New Zealand hospital).26 The findings suggest that text entry data generally have a high degree of accuracy and that the major deficiency in the quality of text entries is completeness.

A further limitation is that this study was carried out with a particular AIMS in a sample of convenience in a single institution where the system is well established. The results may not be generalizable to other commercially available AIMS and may not apply during the adjustment period immediately following introduction of a new AIMS to an institution. The study could not be blinded.

Observation of research participants may alter their behaviour, a phenomenon known as the Hawthorne effect.27,28 When observation causes participants to behave in a manner they believe will be viewed favourably by others, it is known as a “social desirability bias”29 and reduces the generalizability of research conclusions. The Hawthorne effect has been documented in medicine, where the act of observation alone can increase the outcomes of participants in the control arm of RCTs.30,31 Nevertheless, few studies have assessed the effect of direct observation on the behaviour of healthcare professionals.32 Professional behaviour may be subject to a strong social desirability bias. Professional expectations of recordkeeping might mean that observation would promote full completion of anesthetic records in this study.

It is difficult to know the extent to which the change between the two study periods reflected the differences in the intensity of the observation and the extent to which it generally reflected increasing familiarity with being in the study and a reduction of its impact. Either way, the message is clear, i.e., taking part in a study can influence participants’ behaviour, and this is an important potential confounder of relevant research.

The finding that the deterioration in record completeness was greater over time in the handwritten record cohort than in the AIMS records may reflect the proportion of items in the AIMS record for which data collection is automatic or forced. The proportion of items dependent solely on anesthesiologists for completion (and subject to any change in professional behaviour due to observation) was therefore less in the AIMS group than in the handwritten record group. Design elements have been shown to facilitate the completeness of data in paper-based clinical records,23,33 but the ability to force recording of required data is unique to automated records.

The AIMS used in this study uses a “hard stop” (a point in the process of creating an anesthetic record where users cannot proceed without entering certain parameters) to force entry of patient demographic data before an anesthetic record can be created; therefore, these items were documented in all AIMS records. Other variations of hard stops require entry of certain intraoperative data before a record can be finalized and printed at the conclusion of a case. The benefit of increased completion achieved with hard stops needs to be balanced against the number of warning messages necessary to enforce record completion. Unduly onerous hard-stop procedures have the potential to reduce user acceptance of AIMS technology and distract from clinical care. Where agreement among users can be reached as to what fields are essential and should be forced, AIMS technology provides a ready means of increasing completeness of key intraoperative data. An important advantage of an AIMS over conventional methods lies in the greater potential of the former for continuous process improvement.

The anesthetic record is an important form of communication between the various staff involved in a patient’s care and an important record for audit and legal purposes. Although there is a lack of studies investigating the direct benefit of an accurately documented anesthetic record for patient safety, the value of a high-quality record in informing postanesthetic care and subsequent anesthesia is widely accepted. In conclusion, in this study, the AIMS records contained more complete intraoperative information as text data compared with the handwritten records. Our results also added to evidence that observational research has the potential to alter the professional behaviour of participants, and this has implications for the generalizability of conclusions from observational studies.