Journal of Medical Systems

, 42:261 | Cite as

Prediction of Incident Delirium Using a Random Forest classifier

  • John P. CorradiEmail author
  • Stephen Thompson
  • Jeffrey F. Mather
  • Christine M. Waszynski
  • Robert S. Dicks
Systems-Level Quality Improvement
Part of the following topical collections:
  1. Systems-Level Quality Improvement


Delirium is a serious medical complication associated with poor outcomes. Given the complexity of the syndrome, prevention and early detection are critical in mitigating its effects. We used Confusion Assessment Method (CAM) screening and Electronic Health Record (EHR) data for 64,038 inpatient visits to train and test a model predicting delirium arising in hospital. Incident delirium was defined as the first instance of a positive CAM occurring at least 48 h into a hospital stay. A Random Forest machine learning algorithm was used with demographic data, comorbidities, medications, procedures, and physiological measures. The data set was randomly partitioned 80% / 20% for training and validating the predictive model, respectively. Of the 51,240 patients in the training set, 2774 (5.4%) experienced delirium during their hospital stay; and of the 12,798 patients in the validation set, 701 (5.5%) experienced delirium. Under-sampling of the delirium negative population was used to address the class imbalance. The Random Forest predictive model yielded an area under the receiver operating characteristic curve (ROC AUC) of 0.909 (95% CI 0.898 to 0.921). Important variables in the model included previously identified predisposing and precipitating risk factors. This machine learning approach displayed a high degree of accuracy and has the potential to provide a clinically useful predictive model for earlier intervention in those patients at greatest risk of developing delirium.


Delirium Prediction Decision support Machine learning Random forest 


Compliance with Ethical Standards

Conflicts of Interest

The authors do not have any conflicts of interest to declare. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

For this type of study formal consent is not required.

Supplementary material

10916_2018_1109_MOESM1_ESM.xlsx (16 kb)
ESM 1 (XLSX 16 kb)
10916_2018_1109_MOESM2_ESM.pdf (255 kb)
ESM 2 (PDF 255 kb)


  1. 1.
    Inouye, S. K., Westendorp, R. G. J., and Saczynski, J. S., Delirium in elderly people. Lancet 383:911–922, 2014. Scholar
  2. 2.
    Inouye, S. K., Rushing, J. T., Foreman, M. D. et al., Does delirium contribute to poor hospital outcomes?: A three-site epidemiologic study. J. Gen. Intern. Med. 13:234–242, 1998. Scholar
  3. 3.
    Ely, E. W., Gautam, S., Margolin, R. et al., The impact of delirium in the intensive care unit on hospital length of stay. Intensive Care Med. 27:1892–1900, 2001. Scholar
  4. 4.
    Witlox, J., Eurelings, L. S. M., De Jonghe, J. F. M. et al., Delirium in Elderly Patients and the Risk of Postdischarge Mortality. JAMA 304:443–451, 2010. Scholar
  5. 5.
    Rudolph, J. L., and Marcantonio, E. R., Postoperative Delirium: Acute change with long-term implications. Anesth. Analg. 112:1202–1211, 2011. Scholar
  6. 6.
    Girard, T. D., Jackson, J. C., Pandharipande, P. P. et al., Delirium as a predictor of long-term cognitive impairment in survivors of critical illness. Crit. Care Med. 38:1513–1520, 2010. Scholar
  7. 7.
    Langan, C., Sarode, D. P., Russ, T. C. et al., Psychiatric symptomatology after delirium: a systematic review. Psychogeriatrics, 2017. Scholar
  8. 8.
    Davis, D. H. J., Muniz-Terrera, G., Keage, H. A. D. et al., Association of Delirium With Cognitive Decline in Late Life. JAMA Psychiatry 74:244, 2017. Scholar
  9. 9.
    Brown, E., and Douglas, V., Moving Beyond Metabolic Encephalopathy: An Update on Delirium Prevention, Workup, and Management. Semin. Neurol. 35:646–655, 2015. Scholar
  10. 10.
    Hshieh, T. T., Yue, J., Oh, E. et al., Effectiveness of Multicomponent Nonpharmacological Delirium Interventions. JAMA Intern. Med. 175:512, 2015. Scholar
  11. 11.
    Lawlor, P. G., and Bush, S. H., Delirium diagnosis, screening and management. Curr Opin Support Palliat Care 8:286–295, 2014. Scholar
  12. 12.
    Inouye, S. K., Van Dyck, C. H., Alessi, C. A. et al., Clarifying confusion: The confusion assessment method: A new method for detection of delirium. Ann. Intern. Med. 113:941–948, 1990. Scholar
  13. 13.
    Waszynski, C., and Petrovic, K., Nurses’ evaluation of the Confusion Assessment Method: a pilot study. J. Gerontol. Nurs. 34:49–56, 2008. Scholar
  14. 14.
    Ely, E. W., Margolin, R., Francis, J. et al., Evaluation of delirium in critically ill patients: validation of the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU). Crit. Care Med. 29:1370–1379, 2001.CrossRefGoogle Scholar
  15. 15.
    Inouye, S. K., Kosar, C. M., Tommet, D. et al., The CAM-S: Development and validation of a new scoring system for delirium severity in 2 cohorts. Ann. Intern. Med. 160:526–533, 2014. Scholar
  16. 16.
    Van Meenen, L. C. C., van Meenen, D. M. P., de Rooij, S. E., and ter Riet, G., Risk prediction models for postoperative delirium: A systematic review and meta-analysis. J. Am. Geriatr. Soc. 62:2383–2390, 2014. Scholar
  17. 17.
    van den Boogaard, M., Pickkers, P., Slooter, A. J. C. et al., Development and validation of PRE-DELIRIC (PREdiction of DELIRium in ICu patients) delirium prediction model for intensive care patients: observational multicentre study. BMJ 344:e420–e420, 2012. Scholar
  18. 18.
    Kim MY, Park UJ, Kim HT, Cho WH (2016) DELirium Prediction Based on Hospital Information (Delphi) in General Surgery Patients. 95:1–7. doi: CrossRefGoogle Scholar
  19. 19.
    Kennedy, M., Enander, R. A., Tadiri, S. P. et al., Delirium risk prediction, healthcare use and mortality of elderly adults in the emergency department. J. Am. Geriatr. Soc. 62, 2014. Scholar
  20. 20.
    Breiman, L., Random Forests. Mach. Learn. 45:5–32, 2001. Scholar
  21. 21.
    Quan, H., Li, B., Couris, C. M. et al., Updating and validating the charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries. Am. J. Epidemiol. 173:676–682, 2011. Scholar
  22. 22.
    Wasey JO (2016) ICD: Tools for Working with ICD-9 and ICD-10 Codes, and Finding ComorbiditiesGoogle Scholar
  23. 23.
    Team TH a. (2017) h2o: R Interface for H2OGoogle Scholar
  24. 24.
    Chen C, Liaw A, Breiman L (2004) Using random forest to learn imbalanced data. Univ California, Berkeley 1–12. doi: Scholar
  25. 25.
    Khalilia, M., Chakraborty, S., and Popescu, M., Predicting disease risks from highly imbalanced data using random forest. BMC Med Inform Decis Mak 11:51, 2011. Scholar
  26. 26.
    Platt, J., Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Adv Large Margin Classif 10:61–74, 1999. Scholar
  27. 27.
    Kuhn M (2008) Building Predictive Models in R Using the caret Package. J Stat Software, Artic 28:1–26.
  28. 28.
    Robin, X., Turck, N., Hainard, A. et al., pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 12:77, 2011. Scholar
  29. 29.
    Grau, J., Grosse, I., and Keilwagen, J., PRROC: computing and visualizing precision-recall and receiver operating characteristic curves in R. Bioinformatics 31:2595–2597, 2015. Scholar
  30. 30.
    Boulesteix, A.-L., Janitza, S., Kruppa, J., and König, I. R., Overview of random forest methodology and practical guidance with emphasis on computational biology and bioinformatics. Wiley Interdiscip Rev Data Min Knowl Discov 2:493–507, 2012. Scholar
  31. 31.
    Semancik, L., Waszynski, C., and Udeh, E., Delirium in hospitalized patients: recognition, prevention, and management. Conn. Med. 78:105–109, 2014.PubMedGoogle Scholar
  32. 32.
    Tieges, Z., McGrath, A., Hall, R. J., and MacLullich, A. M. J., Abnormal Level of Arousal as a Predictor of Delirium and Inattention: An Exploratory Study. Am. J. Geriatr. Psychiatry 21:1244–1253, 2013. Scholar
  33. 33.
    Han, J. H., Vasilevskis, E. E., Schnelle, J. F. et al., The Diagnostic Performance of the Richmond Agitation Sedation Scale for Detecting Delirium in Older Emergency Department Patients. Acad. Emerg. Med. 22:878–882, 2015. Scholar
  34. 34.
    Morandi, A., Han, J. H., Meagher, D. et al., Detecting Delirium Superimposed on Dementia: Evaluation of the Diagnostic Performance of the Richmond Agitation and Sedation Scale. J. Am. Med. Dir. Assoc. 17:828–833, 2016. Scholar
  35. 35.
    Corradi, J. P., Chhabra, J., Mather, J. F. et al., Analysis of multi-dimensional contemporaneous EHR data to refine delirium assessments. Comput. Biol. Med. 5799:1–24, 2016. Scholar
  36. 36.
    Gulshan, V., Peng, L., Coram, M. et al., Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. Jama 304:649–656, 2016. Scholar
  37. 37.
    Dawes, T. J. W., de Marvao, A., Shi, W. et al., Machine Learning of Three-dimensional Right Ventricular Motion Enables Outcome Prediction in Pulmonary Hypertension: A Cardiac MR Imaging Study. Radiology 161315, 2017. Scholar
  38. 38.
    Ghose, S., Mitra, J., Khanna, S., and Dowling, J., An Improved Patient-Specific Mortality Risk Prediction in ICU in a Random Forest Classification Framework. Stud Health Technol Inform 214:56–61, 2015. Scholar
  39. 39.
    Futoma, J., Morris, J., and Lucas, J., A comparison of models for predicting early hospital readmissions. J. Biomed. Inform. 56:229–238, 2015. Scholar
  40. 40.
    Kessler, R. C., Warner, C. H., Ivany, C. et al., Predicting Suicides After Psychiatric Hospitalization in US Army Soldiers: The Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Psychiatry 72:1–9, 2014. Scholar
  41. 41.
    Mortazavi, B. J., Downing, N. S., Bucholz, E. M. et al., Analysis of Machine Learning Techniques for Heart Failure Readmissions. Circ Cardiovasc Qual Outcomes 9:629–640, 2016. Scholar
  42. 42.
    Taylor, R. A., Pare, J. R., Venkatesh, A. K. et al., Prediction of In-hospital Mortality in Emergency Department Patients with Sepsis: A Local Big Data-Driven, Machine Learning Approach. Acad. Emerg. Med. 23:269–278, 2016. Scholar
  43. 43.
    Kramer, D., Veeranki, S., Hayn, D., and Quehenberger, F., Development and Validation of a Multivariable Prediction Model for the Occurrence of Delirium in Hospitalized Gerontopsychiatry and Internal Medicine Patients. Stud Health Technol Inform:32–39, 2017.
  44. 44.
    Wong, A., Young, A. T., Liang, A. S. et al., Development and Validation of an Electronic Health Record–Based Machine Learning Model to Estimate Delirium Risk in Newly Hospitalized Patients Without Known Cognitive Impairment. JAMA Netw. Open 1:e181018, 2018. Scholar
  45. 45.
    Halladay, C. W., Sillner, A. Y., and Rudolph, J. L., Performance of Electronic Prediction Rules for Prevalent Delirium at Hospital Admission. JAMA Netw. Open 1:e181405, 2018. Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Research DepartmentHartford HospitalHartfordUSA
  2. 2.Division of Geriatric MedicineHartford HospitalHartfordUSA

Personalised recommendations