Abstract
Purpose
To compare the prevalence, characteristics, drug treatment for delirium, and outcomes of patients with Natural Language Processing (NLP) diagnosed behavioral disturbance (NLP-Dx-BD) vs Confusion Assessment Method for intensive care unit (CAM-ICU) positivity.
Methods
In three combined medical-surgical ICUs, we obtained data on demographics, treatment with antipsychotic medications, and outcomes. We applied NLP to caregiver progress notes to diagnose behavioral disturbance and analyzed simultaneous CAM-ICU.
Results
We assessed 2313 patients with a median lowest Richmond Agitation-Sedation Scale (RASS) score of − 2 (− 4.0 to − 1.0) and median highest RASS score of 1 (0 to 1). Overall, 1246 (53.9%) patients were NLP-Dx-BD positive (NLP-Dx-BDpos) and 578 (25%) were CAM-ICU positive (CAM-ICUpos). Among NLP-Dx-BDpos patients, 539 (43.3%) were also CAM-ICUpos. In contrast, among CAM-ICUpos patients, 539 (93.3%) were also NLP-Dx-BDpos. The use of antipsychotic medications was highest in patients in the CAM-ICUpos and NLP-Dx-BDpos group (24.3%) followed by the CAM-ICUneg and NLP-Dx-BDpos group (10.5%). In NLP-Dx-BDneg patients, antipsychotic medication use was lower at 5.1% for CAM-ICUpos and NLP-Dx-BDneg patients and 2.3% for CAM-ICUneg and NLP-Dx-BDneg patients (overall P < 0.001). Regardless of CAM-ICU status, after adjustment and on time-dependent Cox modelling, NLP-Dx-BD was associated with greater antipsychotic medication use. Finally, regardless of CAM-ICU status, NLP-Dx-BDpos patients had longer duration of ICU and hospital stay and greater hospital mortality (all P < 0.001).
Conclusion
More patients were NLP-Dx-BD positive than CAM-ICU positive. NLP-Dx-BD and CAM-ICU assessment describe partly overlapping populations. However, NLP-Dx-BD identifies more patients likely to receive antipsychotic medications. In the absence of NLP-Dx-BD, treatment with antipsychotic medications is rare.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Natural language processing (NLP) of electronic caregiver notes is a novel and powerful epidemiological tool to identify behavioral disturbance in critically ill patients. NLP identifies more patients with abnormal behavior and disturbed cognitive state, who will receive antipsychotic medications, are more severely ill, likely to stay in intensive care unit (ICU) and hospital for longer, and more likely to die than CAM-ICU positive patients. |
Introduction
Delirium is common in critically ill patients [1] and associated with poor outcomes [2, 3]. The fourth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM IV) defines delirium as a “disturbance in consciousness that is accompanied by a change in cognition that cannot be accounted for by a preexisting or evolving dementia” [4]. The manual describes three criteria that may be used to identify such a disturbance in consciousness; A, “reduced clarity of awareness of the environment; B, “accompanying change in cognition” or “development of a perceptual disturbance” and C, “develops over a short period of time and tends to fluctuate during the course of the day”.
The fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM V) reconsiders delirium diagnosis listing five diagnostic criteria. Three describe disturbance in behavior (i.e., “disturbance in attention”, “develops over a short period” and “disturbance in cognition”), and two describe the absence of pre-existing disturbances or conditions (i.e., “preexisting, established or evolving neurocognitive disorder” and “a direct physiological consequence of another medical condition”) that, if present, would exclude a diagnosis of delirium [5]. Although, more restrictive than the DSM IV and with dependences on interpretation, the DSM V criteria have been found to identify a similar population of patients to the DSM IV criteria [6, 7].
Despite such constructs, the systematic identification of delirium in intensive care unit (ICU) patients through direct application of these criteria is considered challenging [8]. As a consequence, derivative methodologies have been developed to operationalize screening for delirium in the ICU [9]. Of these, the Confusion Assessment Method for ICU (CAM-ICU) [10, 11] and the Intensive Care Delirium Screening Checklist (ICDSC) [12], both derived from the DSM IV, have been recommended as the most reliable for use within the adult ICU [13]. In our ICUs, the CAM-ICU methodology is used by bedside nurses once per shift to assess patients for possible delirium. The results of this assessment are documented in the electronic progress notes. However, progress notes also contain nursing, medical, and allied health caregivers’ narrative observations, which may further describe a patient’s behavior and cognitive state.
We hypothesized that, in their clinical progress notes, caregivers might use words suggestive of disturbed behavior or cognition such as “agitated” and “confused”, which would increase the information available for the assessment of the patient’s cognitive state. In a recent study [14], we successfully used Natural Language Processing (NLP) techniques to detect these words in the electronic clinical progress notes of a large cohort of critically ill patients. NLP is a software technology that is used to automatically analyze textual information in documentation such as progress notes. We referred to patients identified by such words as having “NLP-Diagnosed Behavioural Disturbance” (NLP-Dx-BD).
Accordingly, in this study, we compared the prevalence, clinical characteristics, outcomes, and antipsychotic medications-based treatment of patients with NLP-Dx-BD with those of patients with CAM-ICU-based delirium screening assessments.
Methods
Study design
This was a non-interventional, retrospective study, which used data derived from the electronic health records (EHR) of a university affiliated ICU system in Melbourne Australia. This study was approved by the Austin Hospital Human Research Ethics Committee (LNR/19/Austin/38), which waived the need for informed consent. We included all adult patients (≥ 18 years old) admitted to the ICUs of the Austin Hospital, Melbourne between 1 may 2019 and 31 December 2020. If a patient had multiple admissions only the first admission was considered for inclusion. No additional exclusion criteria were applied.
Data collection and manipulation
All baseline and outcome data were collected from the Australian and New Zealand Intensive Care Society Adult ICU Patient Database run by the Centre for Outcome and Resource Evaluation [15]. Using a proprietary intensive care clinical information system, we obtained electronic data from all progress notes entered into the ICU-specific electronic health record (EHR) by clinical staff including doctors, nurses, physiotherapists and other allied health professionals.
During the study period, by ICU policy, patients received general care aimed at decreasing the risk of delirium, including frequent family visits, dimmed lights at night, minimal interaction to facilitate night-time sleep cycling, and ensuring use of spectacles and hearing aids as necessary. CAM-ICU and Richmond Agitation-Sedation Scale (RASS) assessments were recorded in the clinical progress notes by nursing staff during every 8-h shift. NLP techniques were then used to extract the detail of these assessments from the notes. Further, the clinical progress notes of all caregivers were analyzed using NLP tokenizing techniques (natural language toolkit; NLTK 3.5) [16]. As previously described, [14] progress notes were converted to sentence vectors. each vector was then searched for the presence of words, terms, or expressions, suggestive of behavioral disturbance in accordance with terms selected in a previously published survey of clinical staff [17] (etable 1a and 1b in online supplement). In this survey, clinical staff were asked to identify words, terms or expressions that they would use to describe a situation where they thought a patient was exhibiting disturbed behavior.
Finally, we obtained data on antipsychotic medications used in our ICUs (haloperidol, olanzapine, quetiapine, and risperidone) from the hospital EHR and correlated them with other study data. We hypothesized that treatment with antipsychotic medications may reflect care givers discomfort with patients cognitive dysfunction and disturbed behavior. Accordingly, we chose this outcome as the clinically relevant primary outcome for our study.
Data definition
We searched nursing progress notes for CAM-ICU and RASS assessments using NLP. Simultaneously, we analyzed medical, nursing and allied health progress notes for words describing NLP-Dx-BD.
We classified patients as CAM-ICU positive, negative, or unable to be assessed (e.g. the patient does not speak English or the patient refuses to cooperate) or missing.
We then classified patients as CAM-ICU positive when, in at least one progress note, a CAM-ICU positive assessment was reported.
We classified patients as NLP-Dx-BD positive when, in at least one progress note, we found a word indicative of behavioral disturbance (eTable 1a and eTable 1b).
Exposure
The primary exposure of the present study was the combination of CAM-ICU and NLP-Dx-BD diagnoses. Patients were, therefore, classified in four groups:
Group 1: Negative for both assessments.
Group 2: Only CAM-ICU positive.
Group 3: Only NLP-Dx-BD positive.
Group 4: Positive for both assessments.
Outcomes
The primary outcome was use of antipsychotic medications. Such drugs included haloperidol, quetiapine, olanzapine and risperidone. Despite controversy [18], these medications represent the most frequent medical intervention applied to agitation, perceived delirium and/or behavioral disturbance in ICU.
Secondary outcomes included the duration of mechanical ventilation (in ventilated patients), ICU and hospital length of stay, and ICU, hospital, and 28-day in-hospital mortality.
Statistical analysis
All continuous data are reported as medians (quartile 25%–quartile 75%) and categorical data as numbers and percentages. Baseline, clinical characteristics, and outcomes of the patients were compared among the groups using Fisher exact test and the Kruskal–Wallis test. The proportion of patients receiving medications over time is presented in Kaplan–Meier curves and compared with the log-rank test. Multivariable logistic regression models were used to assess the association of the exposure groups with hospital mortality, with Group 1 (CAM-ICU negative and NLP-Dx-BD negative) used as reference. To account for immortal time bias, we additionally conducted a time-dependent Cox proportional hazard model for the primary outcome and hospital mortality that considered all measurements available in each note. For the primary outcome only exposures happening before the first outcome (first time the patient received an antipsychotic) were included in the model to avoid exposures measured after the medication had already been given. The models were adjusted by age, type of admission, and by the Australian and New Zealand Risk of Death (ANZROD) after log transformation [19]. ANZROD is a validated and accurate predictor of mortality in ICUs in Australia and New Zealand [20]. Effect estimates were reported as hazard ratio (OR) with its 95% confidence interval (CI).
Sensitivity analyses were performed stratifying the cohort according to the use of mechanical ventilation. Overall rate of missing data was low, and is reported in the eTable 2 in the Online Supplement. All analyses were case complete analysis and were conducted in R v.4.0.3 (R Foundation) [21] and a P value < 0.05 was considered significant.
Results
The baseline characteristics of the study patients according to the four groups are shown in Table 1. Their median age was 63 years, most were male, most were admitted due to medical conditions, and the largest group of patients were admitted due to cardiovascular disease. The most prevalent pre-existing disorder was diabetes, followed by chronic lung disease. Overall, just above half received mechanical ventilation and just below half received vasopressor/inotropic drugs. However, irrespective of CAM-ICU status, the presence of NLP-Dx-BD positivity identified patients with greater illness severity. The mean RASS during ICU admission was − 0.5 (− 1 to − 0.2), and was similar between the groups (P = 0.238) (eFigure 1 in Online Supplement).
Characteristics of CAM-ICU delirium and NLP behavioral disturbance diagnosis
The time to first CAM-ICU positivity or to first NLP-Dx-BD assessment is shown in eFigure 2 in the Online Supplement. The frequency of words used to describe NLP-Dx-BD is show in eTable 3 and, as word cloud, in eFigure 3 in the Online Supplement. Overall, 32% of the patients with NLP-Dx-BD were identified on day 0 (day of admission), but only 11% of CAM-ICU positive patients were identified on this day. In addition, the majority of patients positive for NLP-Dx-BD (37%) and of CAM-ICU (29%) were identified on day 1.
Among NLP-Dx-BD-positive patients, the median number of notes with BD positive words was 3 (1–8) and the median cumulative number of BD positive words was 4 (2–13) per patient. Among CAM-ICU positive patients, the median number of notes that were positive for NLP-Dx-BD was 7 (3–18) and the median cumulative number of words was 11 (4–30). The median number of ICU-shift notes per patient where CAM-ICU was reported was 3 (2–7). Among CAM-ICU positive patients a median of 2 (1–4) notes were positive. The number of notes per patient with CAM-ICU reported as “unable to be done” was 0 (0–2).
Classification according to NLP-Dx-BD and CAM-ICU
We studied 2932 patients. Among these, NLP-Dx-BD or CAM-ICU assessments were not available and reported as “unable to be performed” or missing in 1 (0%) and in 619 (21.1%) patients, respectively, leaving 2313 patients with complete data for analysis. Of these, 1246 (53.9%) were NLP-Dx-BD positive and 578 (25%) were CAM-ICU positive. Among NLP-Dx-BD positive patients 539 (43.3%) were CAM-ICU positive, while among CAM-ICU positive patients 539 (93.3%) were NLP-Dx-BD positive.
When assessing the four key groups, 1028 (44.4%) were categorized in Group 1; 39 (1.7%) in Group 2; 707 (30.6%) in Group 3; and 539 (23.3%) in Group 4. The distribution of patients into these groups shows that NLP-Dx-BD identified more patients than CAM-ICU (Fig. 1). It also demonstrates that, overall, only 44% of patients were negative for both assessments during their ICU stay.
We additionally assessed categorization according to different levels of conscious state from coma to agitation using the lowest and highest RASS score (eFigure 4 and eFigure 5, Online Supplement) to illustrate the impact of such states during each shift on the classification of patients according to NLP-Dx-BD and CAM-ICU status. NLP identified more patients than CAM-ICU when the shift RASS score was in the awake range and when the highest shift RASS score was in the coma or agitated range.
Primary outcome: use of antipsychotic medications
The use of antipsychotic medications was highest in patients in Group 4 (24.3%) (Table 2 and Fig. 2). The group with the second greatest use of these medications was Group 3 (10.5%). For all antipsychotic medications, the rate of use in Group 2 patients was 5.1%, similar to that of Group 1 at 2.3%. (Table 2). The impact of NLP-Dx-BD on the use of antipsychotic medications was confirmed by univariate and multivariate Cox models that treated groups as time-dependent variables (eTable 4 and eTable 5).
By day 3, most of such treatment had been given, with the exception of Group 4 patients who continued to accrue further treatment up to one week after ICU admission (Fig. 3).
Secondary clinical outcomes
Clinical outcomes of study patients are shown in Table 3. Overall hospital mortality was 6.3%. Mortality was highest in Group 4 (10.8%), lower but similar in Group 2 and Group 3 (7.7% and 7.5%, respectively), and lowest in Group 1. Duration of ICU and hospital stay was significantly longer in the presence of NLP-Dx-BD positivity, irrespective of CAM-ICU status.
On univariable analysis, both Group 4 (OR, 3.77 [95% CI 2.43–5.95]; P < 0.001) and Group 3 (OR, 2.53 [95% CI, 1.62–4.00]; P < 0.001) were associated with increased risk for hospital mortality (eTable 6 in Online Supplement). On multivariable analysis, only Group 3 (OR 1.69 [95% CI 1.05–2.76]; P = 0.03) was independently associated with increased risk for hospital mortality (eTable 7). On univariate modelling that treated groups as time-dependent variables both NLP-Dx-BD and CAM-ICU status were associated with mortality but, on multivariable modelling, no overall effect was found (eTable 8 and eTable 9).
Discussion
Key findings
In a cohort of more than two thousand ICU patients, we compared the prevalence, characteristics, treatment with antipsychotic medications, and outcomes of patients with NLP-diagnosed behavioral disturbance (NLP-Dx-BD) and patients with Confusion Assessment Method for ICU (CAM-ICU) positivity. We found that more than half were NLP-Dx-BD positive and a quarter were CAM-ICU positive. Among NLP-Dx-BD positive patients, four out of ten were CAM-ICU positive. In contrast, among CAM-ICU positive patients, nine out of ten were NLP-Dx-BD positive. NLP-Dx-BD identified significantly more patients likely to receive antipsychotic medications and, in the absence of NLP-Dx-BD, treatment with antipsychotic medications was uncommon. Finally, regardless of CAM-ICU status, NLP-Dx-BD was associated with significantly longer duration of ICU and hospital stay and greater hospital mortality (Table 4).
Relationship to previous studies
Dependent on risk factors and delirium phenotype, the prevalence of CAM-ICU positive patients has been reported between 10 and 89% [22,23,24,25]. Our prevalence of 25% falls within this range. CAM-ICU positive patients appear to have a hospital mortality rate of between 10.7 and 27% [26,27,28]. Our hospital mortality rate of 10.6% falls just below this range. Further, our finding that NLP-Dx-BD identified more patients with behavioral and/or cognitive disturbance earlier in the critical care episode than CAM-ICU is also consistent with previous findings that, in routine practice, the use of CAM-ICU may delay the detection of delirium [29]. Our observations that the prevalence of behavioral disturbance as detected by CAM-ICU was lower than with NLP-Dx-BD are also similar to findings from previous studies where CAM-ICU was found to detect lower rates of delirium than unstructured assessments [29, 30].
CAM-ICU was not obtained in one in five patients. This is aligned with work by Terry et al. who found a missed documentation rate of more than half of available opportunities in a pair of American medical and surgical ICUs [31] and with data from Vanderbilt University Medical Centre showing a 16% non-compliance for CAM-ICU assessment in an institution highly dedicated to CAM-ICU assessment [32]. Moreover, Kanova et al. reported in a prospective study that 14% of patients could not be assessed for CAM-ICU due to prolonged coma [33]. Our observations indicate that, during a given shift, and in the presence of RASS defined coma, CAM-ICU assessment was missing in most patients. However, this was also true to a similar extent for NLP-Dx-BD words and coma was an uncommon state in our cohort.
Finally, our observations of the prevalence, characteristics, and outcome of NLP-Dx-BD are aligned with those found in a much larger population of more than 12,000 patients recently investigated by our group [14]. Moreover, the words used to define this condition are supported by our survey of the terms that clinicians would use to describe the presence of an acute behavioral disturbance [17].
Implications of study findings
Our findings imply that NLP-Dx-BD may identify more patients with abnormal behavioral and disturbed cognitive state than CAM-ICU. Moreover, they imply that NLP-Dx-BD positive patients are significantly more likely to receive antipsychotic medications than CAM-ICU positive patients. Furthermore, they suggest that NLP-Dx-BD positive patients are more severely ill, more likely to stay in ICU and hospital for longer, and more likely to die. This is consistent with a previous study that found that agitated behavior, even without diagnosed delirium, correlated with poorer outcomes [34]. It is also consistent with clinical observations that the level of sedation and behavioral symptoms often fluctuate over an 8-h or 12-h shift. Thus, NLP-Dx-BD will capture behavioral screening features over the whole period but CAM-ICU, which represents a 'snapshot" at a single point of time during the shift, may not. Finally, our findings suggest that NLP may be particularly more sensitive during shifts where a given RASS score has identified the presence of either deep sedation or agitation.
Strengths and limitations
Our study has several strengths. It studied a novel approach to the assessment of the epidemiology of behavioral disturbance in critically ill patients. It compared such assessment with delirium assessment performed by the CAM-ICU methodology. It also conducted such comparisons in a large and heterogeneous group of ICU patients, thus increasing the external validity of the observations. The incidence of CAM-ICU positive assessment was consistent with the literature and our study applied a primary outcome based on therapy, which is likely to be relevant to clinicians as well as patients. Finally, by showing that NLP-Dx-BD positivity captured almost all of patients with CAM-ICU positive status, it provided additional indirect evidence of the validity of its construct.
We acknowledge several limitations. Our study was undertaken in a large tertiary intensive care unit system in a university affiliated hospital of a resource-rich country. Therefore, its findings may not apply to other intensive care units in low or middle-income countries. Multiple progress notes were missing a positive or negative CAM-ICU assessment. However, this may reflect the well-described operational challenges in applying CAM-ICU and had minimal material impact on our findings. We did not provide data on non-pharmacological interventions including family visits and environmental management. However, many were applied by unit policy and the efficacy of such interventions remains uncertain as demonstrated in a recent stepped wedge cluster controlled trial [35]. Patients were not assessed for the presence of delirium by an independent, psychiatrically trained clinician. However, we were studying the relationship between patient populations identified through the use of alternative screening methodologies. Thus, we were not attempting to confirm that the patients identified by such screening would then be diagnosed with delirium by such clinicians. Therefore, the changes in behavior may have reflected drug withdrawal, exacerbation of dementia or of pre-existing psychiatric disorders. Care givers generating the progress notes may have been aware of CAM-ICU results at the time of writing their notes; however, it is impossible to determine if an NLP-Dx-BD positive note preceded or followed a CAM-ICU assessment. Further, significantly more patients were NLP-Dx-BD positive than CAM-ICU positive, thus demonstrating that NLP-Dx-BD positivity was often logically determined independently of CAM-ICU assessment. Clinical progress notes may include errors in annotation that incorrectly characterize a patient’s behavior [36, 37]. However, our study analyzed multiple progress notes (nursing, medical, allied health) that documented the same shift making systematic errors unlikely. There may also have been errors in the records of administration of anti-psychotic medication. However, our ICU uses an audited medication management system that accounts for the prescription, retrieval and administration of medications, which would minimize such errors. Finally, we did not assess long term cognitive outcomes. Thus we cannot make any statement about the long term sequelae of NLP-Dx-BD.
Conclusion
In conclusion, Natural Language Processing of electronic caregiver notes and the Confusion Assessment Method for ICU (CAM-ICU) describe partly overlapping populations. However, NLP-Dx-BD identified more patients with behavioral and/or cognitive disturbance than CAM-ICU, while identifying more than 90% of CAM-ICU positive patients as having such a disturbance. Moreover, NLP-Dx-BD identified significantly more patients who went on to receive treatment with antipsychotic medications. In contrast, in the absence of NLP-Dx-BD, very few CAM-ICU positive patients received such treatment. Further, irrespective of CAM-ICU status, NLP-Dx-BD also identified patients with longer duration of ICU and hospital stay and greater hospital mortality. These observations suggest that NLP-Dx-BD may be a novel and useful epidemiologic screening tool in ICU patients.
References
van den Boogaard M, Schoonhoven L, van der Hoeven JG et al (2012) Incidence and short-term consequences of delirium in critically ill patients: a prospective observational cohort study. Int J Nurs Stud 49:775–783. https://doi.org/10.1016/j.ijnurstu.2011.11.016
Salluh JIF, Wang H, Schneider EB et al (2015) Outcome of delirium in critically ill patients: systematic review and meta-analysis. BMJ 350:h2538–h2538. https://doi.org/10.1136/bmj.h2538
Cole MG, Ciampi A, Belzile E, Zhong L (2008) Persistent delirium in older hospital patients: a systematic review of frequency and prognosis. Age Ageing 38:19–26. https://doi.org/10.1093/ageing/afn253
American Psychiatric Association (1998) Diagnostic and statistical manual of mental disorders: DSM-IV ; includes ICD-9-CM codes effective 1. Oct. 96, 4. ed., 7. print. Washington, DC
Association AP, Association AP (2013) Diagnostic and statistical manual of mental disorders: DSM-5, 5th edn. American Psychiatric Association, Washington, D.C.
Adamis D, Rooney S, Meagher D et al (2015) A comparison of delirium diagnosis in elderly medical inpatients using the CAM, DRS-R98, DSM-IV and DSM-5 criteria. Int Psychogeriatr 27:883–889. https://doi.org/10.1017/S1041610214002853
Meagher DJ, Morandi A, Inouye SK et al (2014) Concordance between DSM-IV and DSM-5 criteria for delirium diagnosis in a pooled database of 768 prospectively evaluated patients using the delirium rating scale-revised-98. BMC Med 12:164. https://doi.org/10.1186/s12916-014-0164-8
Kotfis K, Marra A, Ely EW (2018) ICU delirium—a diagnostic and therapeutic challenge in the intensive care unit. 50:13
Neufeld KJ, Nelliot A, Inouye SK et al (2014) Delirium diagnosis methodology used in research: a survey-based study. Am J Geriatr Psychiatry 22:1513–1521. https://doi.org/10.1016/j.jagp.2014.03.003
Ely EW, Inouye SK, Bernard GR et al (2001) Delirium in mechanically ventilated patients: validity and reliability of the confusion assessment method for the intensive care unit (CAM-ICU). JAMA 286:2703. https://doi.org/10.1001/jama.286.21.2703
Inouye SK (1990) Clarifying confusion: the confusion assessment method: a new method for detection of delirium. Ann Intern Med 113:941. https://doi.org/10.7326/0003-4819-113-12-941
Bergeron N, Dubois M-J, Dumont M et al (2001) Intensive care delirium screening checklist: evaluation of a new screening tool. Intensive Care Med 27:859–864. https://doi.org/10.1007/s001340100909
Barr J, Fraser GL, Puntillo K et al (2013) Clinical practice guidelines for the management of pain, agitation, and delirium in adult patients in the intensive care unit. Crit Care Med 41:263–306. https://doi.org/10.1097/CCM.0b013e3182783b72
Young M, Holmes N, Robbins R et al (2021) Natural language processing to assess the epidemiology of delirium-suggestive behavioural disturbances in critically ill patients. Crit Care Resusc 23:145–154
Stow PJ, Hart GK, Higlett T et al (2006) Development and implementation of a high-quality clinical database: the Australian and New Zealand Intensive Care Society Adult Patient Database. J Crit Care 21:133–141. https://doi.org/10.1016/j.jcrc.2005.11.010
Bird S, Klein E, Loper E (2009) Natural language processing with Python. O’Reilly, Newton
Holmes NE, Amjad S, Young M et al (2019) Using language descriptors to recognise delirium: a survey of clinicians and medical coders to identify delirium-suggestive words. Crit Care Resusc 21:4
Nikooie R, Neufeld KJ, Oh ES et al (2019) Antipsychotics for treating delirium in hospitalized adults: a systematic review. Ann Intern Med 171:485. https://doi.org/10.7326/M19-1860
Paul E, Bailey M, Pilcher D (2013) Risk prediction of hospital mortality for adult patients admitted to Australian and New Zealand intensive care units: development and validation of the Australian and New Zealand Risk of Death model. J Crit Care 28:935–941. https://doi.org/10.1016/j.jcrc.2013.07.058
Eldho P, Bailey M, Kasza J, Pilcher D (2016) The ANZROD model: better benchmarking of ICU outcomes and detection of outliers. Crit Care Resusc 18:12
R Core Team (2019) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna
Devlin JW, Fraser GL, Joffe AM et al (2013) The accurate recognition of delirium in the ICU: the emperor’s new clothes? Intensive Care Med 39:2196–2199. https://doi.org/10.1007/s00134-013-3105-7
Agarwal V, Oneill PJ, Cotton BA et al (2010) Prevalence and risk factors for development of delirium in burn intensive care unit patients. J Burn Care Res 31:706–715. https://doi.org/10.1097/BCR.0b013e3181eebee9
Guenther U, Popp J, Koecher L et al (2010) Validity and Reliability of the CAM-ICU Flowsheet to diagnose delirium in surgical ICU patients. J Crit Care 25:144–151. https://doi.org/10.1016/j.jcrc.2009.08.005
Haenggi M, Blum S, Brechbuehl R et al (2013) Effect of sedation level on the prevalence of delirium when assessed with CAM-ICU and ICDSC. Intensive Care Med 39:2171–2179. https://doi.org/10.1007/s00134-013-3034-5
Tomasi CD, Grandi C, Salluh J et al (2012) Comparison of CAM-ICU and ICDSC for the detection of delirium in critically ill patients focusing on relevant clinical outcomes. J Crit Care 27:212–217. https://doi.org/10.1016/j.jcrc.2011.05.015
Yamamoto T, Mizobata Y, Kawazoe Y et al (2020) Incidence, risk factors, and outcomes for sepsis-associated delirium in patients with mechanical ventilation: a sub-analysis of a multicenter randomized controlled trial. J Crit Care 56:140–144. https://doi.org/10.1016/j.jcrc.2019.12.018
Pauley E, Lishmanov A, Schumann S et al (2015) Delirium is a robust predictor of morbidity and mortality among critically ill patients treated in the cardiac intensive care unit. Am Heart J 170:79-86.e1. https://doi.org/10.1016/j.ahj.2015.04.013
van Eijk MM, van den Boogaard M, van Marum RJ et al (2011) Routine use of the confusion assessment method for the intensive care unit: a multicenter study. Am J Respir Crit Care Med 184:340–344. https://doi.org/10.1164/rccm.201101-0065OC
Reade MC, Eastwood GM, Peck L (2011) Routine use of the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU) by bedside nurses may underdiagnose delirium. Crit Care Resusc 13:9
Terry KJ, Anger KE, Szumita PM (2015) Prospective evaluation of inappropriate unable-to-assess CAM-ICU documentations of critically ill adult patients. J Intensive Care 3:52. https://doi.org/10.1186/s40560-015-0119-y
Soja SL, Pandharipande PP, Fleming SB et al (2008) Implementation, reliability testing, and compliance monitoring of the Confusion Assessment Method for the Intensive Care Unit in trauma patients. Intensive Care Med 34:1263–1268. https://doi.org/10.1007/s00134-008-1031-x
Kanova M, Sklienka P, Roman K et al (2017) Incidence and risk factors for delirium development in ICU patients—a prospective observational study. Biomed Pap Med Fac Univ Palacky Olomouc Czechoslov 161:187–196. https://doi.org/10.5507/bp.2017.004
Marquis F, Ouimet S, Riker R et al (2007) Individual delirium symptoms: Do they matter?*. Crit Care Med 35:2533–2537. https://doi.org/10.1097/01.CCM.0000284506.43390.F3
Rood PJT, Zegers M, Ramnarain D et al (2021) The Impact Of Nursing Delirium Preventive Interventions in the ICU: a multicenter cluster-randomized controlled clinical trial. Am J Respir Crit Care Med 204:682–691. https://doi.org/10.1164/rccm.202101-0082OC
Ratnapalan S, Brown K, Cieslak P et al (2012) Charting errors in a teaching hospital. Pediatr Emerg Care 28:268–271. https://doi.org/10.1097/PEC.0b013e3182494fb8
Carroll AE, Tarczy-Hornoch P, O’Reilly E, Christakis DA (2003) Resident documentation discrepancies in a neonatal intensive care unit. Pediatrics 111:976–980. https://doi.org/10.1542/peds.111.5.976
Acknowledgements
The authors wish to acknowledge the contribution of Michael Reade, Raghaven Murugan, Antonio Paulo Nassar Junior, Yahya Shehabi, Lakhmir Chawla and Michele Gaca for their review of this manuscript and subsequent guidance.
Funding
Open Access funding enabled and organized by CAUL and its Member Institutions.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest
The authors declare they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which permits any non-commercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc/4.0/.
About this article
Cite this article
Young, M., Holmes, N., Kishore, K. et al. Natural language processing diagnosed behavioral disturbance vs confusion assessment method for the intensive care unit: prevalence, patient characteristics, overlap, and association with treatment and outcome. Intensive Care Med 48, 559–569 (2022). https://doi.org/10.1007/s00134-022-06650-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00134-022-06650-z