Abstract
Machine learning can be considered as the current gold standard for predicting deterioration in Intensive Care Unit patients and is in extensive use throughout the world in different fields. As confirmed by many studies, preventing the occurrence of the onset of deterioration in a sufficient time window is a priority in healthcare centers. Also, the significance of enhancing the quality of hospital care and the reduction of adverse outcomes is of great importance. Notably, it is hypothesized that by exploiting recent technologies, models built upon dynamic variables (e.g. vital signs, lab tests, and demographic variables) could reinforce the predictive ability of models aimed at detection of in clinical deterioration with high accuracy, sensitivity and specificity. This manuscript summarises the techniques and approaches proposed in the literature for predicting deterioration and compares the performance and limitations of various approaches grouped based on their application. While several approaches can attain promising results, there is still room for additional improvement, especially in pre-processing and modeling enhancement steps where most methods do not take the necessary steps for ensuring a high-performance result. In this manuscript, the most effective machine learning models, as well as deep learning models, for predicting deterioration of patients are discussed in hopes of assisting the readers with ascertaining the best possible solutions for this problem.
This is a preview of subscription content, access via your institution.



References
AlNuaimi, Noura, Mohammad M Masud, and Farhan Mohammed (2015) ICU patient deterioration prediction: a data-mining approach, arXiv preprint arXiv:1511.06910
Bonafide CP, Russell Localio A, Song L, Roberts KE, Nadkarni VM, Priestley M, Paine CW, Zander M, Lutts M, Brady PW (2014) Cost-benefit analysis of a medical emergency team in a children’s hospital. Pediatrics 134:235–241
Byrd RJ, Steinhubl SR, Sun J, Ebadollahi S, Stewart WF (2014) Automatic identification of heart failure diagnostic criteria, using text analysis of clinical notes from electronic health records. Int J Med Inform 83:983–992
Capan M, Ivy JS, Rohleder T, Hickman J, Huddleston JM (2015) Individualizing and optimizing the use of early warning scores in acute medical care for deteriorating hospitalized patients. Resuscitation 93:107–112
Che Z, Sanjay P, Robinder K, Yan Liu. 2015. Distilling knowledge from deep networks with applications to healthcare domain, arXiv preprint arXiv:1512.03542
Chen L, Ogundele O, Clermont G, Hravnak M, Pinsky MR, Dubrawski AW (2017) Dynamic and personalized risk forecast in step-down units Implications for monitoring paradigms. Ann Am Thorac Soc 14:384–391
Chen Qi, Wang W, Fangyu Wu, De S, Wang R, Zhang B, Huang X (2019) A survey on an emerging area: deep learning for smart city data. IEEE Trans Emerg Top Comput IntelL 3:392–410
Chen X-W, Lin X (2014) Big data deep learning: challenges and perspectives. IEEE Access 2:514–525
Chien I, Alvin S, Alex C, Charlotta L (2018) Identification of serious illness conversations in unstructured clinical notes using deep neural networks. International workshop on artificial intelligence in health. Springer
Churpek MM, Yuen TC, Edelson DP (2013) Predicting clinical deterioration in the hospital: the impact of outcome selection. Resuscitation 84:564–568
Churpek MM, Yuen TC, Park SY, Gibbons R, Edelson DP (2014) Using electronic health record data to develop and validate a prediction model for adverse outcomes on the wards. Crit Care Med 42:841
Churpek MM, Yuen TC, Winslow C, Meltzer DO, Kattan MW, Edelson DP (2016) Multicenter comparison of machine learning methods and conventional regression for predicting clinical deterioration on the wards. Crit Care Med 44:368
Colque RVHM (2018) Robust approaches for anomaly detection applied to video surveillance
Crump C, Sunil S, Bruce W, Patrick F, Azhar R, Christine TS (2009) Using Bayesian networks and rule-based trending to predict patient status in the intensive care unit. In: AMIA Annual Symposium Proceedings, 124, American Medical Informatics Association
Dernoncourt F, Ji YL, Peter S (2017) NeuroNER: an easy-to-use program for named-entity recognition based on neural networks, arXiv preprint arXiv:1705.05487
Desautels T, Calvert J, Hoffman J, Jay M, Kerem Y, Shieh L, Shimabukuro D, Chettipally U, Feldman MD, Barton C (2016) Prediction of sepsis in the intensive care unit with minimal electronic health record data: a machine learning approach. JMIR Med Inform 4:e28
Donald R, Tim H, Ian P, Chambers I, Citerio G, Enblad P, Gregson B, Kiening K, Mattern J, Nilsson P (2012) Early warning of EUSIG-defined hypotensive events using a Bayesian artificial neural network. Intracranial pressure and brain monitoring. Springer
Edelson DP, Carey K, Winslow CJ, Churpek MM (2018) Less is more: detecting clinical deterioration in the hospital with machine learning using only age, heart rate and respiratory rate. C15. Critical care: big data and artificial intelligence in critical illness. American Thoracic Society
Eshelman LJ, Lee KP, Joseph JF, Wei Z, Larry N, Mohammed S (2008) Development and evaluation of predictive alerts for hemodynamic instability in ICU patients. In: AMIA annual symposium proceedings, American Medical Informatics Association. 379
Fawcett T (2006) An introduction to ROC analysis. Pattern Recogn Lett 27:861–874
Forkan AR, Mohammad IK, Atiquzzaman M (2017) ViSiBiD: a learning model for early discovery and real-time prediction of severe clinical events using vital signs as big data. Comput Netw 113:244–257
Fukushima K, Ueno Y, Kawagishi N, Kondo Y, Inoue J, Kakazu E, Ninomiya M, Wakui Y, Saito N, Satomi S (2011) The nutritional index ‘CONUT’is useful for predicting long-term prognosis of patients with end-stage liver diseases. Tohoku J Exp Med 224:215–219
Gao T, Dan G, Matt W, Radford RJ, Alex A (2006) Vital signs monitoring and patient tracking over a wireless network. In: 2005 IEEE engineering in medicine and biology 27th annual conference, IEEE, 102–05
Ghosh S, Li J, Cao L, Ramamohanarao K (2017) Septic shock prediction for ICU patients via coupled HMM walking on sequential contrast patterns. J Biomed Inform 66:19–31
Grandini M, Enrico B, Giorgio V (2020) Metrics for multi-class classification: an overview, arXiv preprint arXiv:2008.05756
Grant S (2018) Limitations of track and trigger systems and the national early warning score. Part 1: areas of contention. Br J Nurs 27:624–631
Han J, Micheline K, Anthony KHT (2001) Spatial clustering methods in data mining. Geographic data mining and knowledge discovery. Taylor & Francis
Harutyunyan H, Hrant K, David CK, Greg VS, Aram G (2017) Multitask learning and benchmarking with clinical time series data, arXiv preprint arXiv:1703.07771
He N, Fang L, Li S, Plaza A, Plaza J (2018) Remote sensing scene classification using multilayer stacked covariance pooling. IEEE Trans Geosci Remote Sens 56:6899–6910
Henriksen DP, Mikkel B, Annmarie TL (2014) Prognosis and risk factors for deterioration in patients admitted to a medical emergency department, PloS one 9
Hogan H, Hutchings A, Wulff J, Carver C, Holdsworth E, Welch J, Harrison D, Black N (2019) Interventions to reduce mortality from in-hospital cardiac arrest: a mixed-methods study. Health Serv Deliv Res 7:1–110
Hoogendoorn M, Ali El H, Kwongyen M, Marzyeh G, Peter S (2016) Prediction using patient comparison vs. modeling: a case study for mortality prediction. In: 2016 38th annual international conference of the IEEE engineering in medicine and biology society (EMBC), IEEE, 2464–67
Hu SB, Wong DJL, Correa A, Li N, Deng JC (2016) Prediction of clinical deterioration in hospitalized adult patients with hematologic malignancies using a neural network model. PLoS ONE 11:e0161401
Huang C-L, Wang C-J (2006) A GA-based feature selection and parameters optimizationfor support vector machines. Expert Syst Appl 31:231–240
Johnson A, Pollard T, Mark R III (2016a) MIMIC-III clinical database. PhysioNet 10:C2XW26
Johnson AEW, Tom JP, Roger GM (2017) Reproducibility in critical care: a mortality prediction case study. In: Machine learning for healthcare conference, 361–76
Johnson AEW, Pollard TJ, Lu Shen H, Li-wei L, Feng M, Ghassemi M, Moody B, Szolovits P, Celi LA, Mark RG (2016) MIMIC-III, a freely accessible critical care database. Sci data 3:160035
Johnson AEW, Stone DJ, Celi LA, Pollard TJ (2017) The MIMIC code repository: enabling reproducibility in critical care research. J Am Med Inform Assoc 25:32–39
Johnson AEW, Stone DJ, Celi LA, Pollard TJ (2018) The MIMIC code repository: enabling reproducibility in critical care research. J Am Med Inform Assoc 25:32–39
Johnson L, Zheng M, Vorobyeva Y, Gabriel A, Qi H, Velásquez N (2016) NMC Horizon report: 2016 higher education edition
Kate RJ, Perez RM, Mazumdar D, Pasupathy KS, Nilakantan V (2016) Prediction and detection models for acute kidney injury in hospitalized older adults. BMC Med Inform Decis Mak 16:39
Kipnis P, Turk BJ, Wulf DA, LaGuardia JC, Liu V, Churpek MM, Romero-Brufau S, Escobar GJ (2016) Development and validation of an electronic medical record-based alert score for detection of inpatient deterioration outside the ICU. J Biomed Inform 64:10–19
Kivipuro M, Tirkkonen J, Kontula T, Solin J, Kalliomäki J, Pauniaho S-L, Huhtala H, Yli-Hankala A, Hoppu S (2018) National early warning score (NEWS) in a Finnish multidisciplinary emergency department and direct vs. late admission to intensive care. Resuscitation 128:164–169
Komorowski M, Leo AC, Omar B, Anthony CG, Aldo AF (2018) The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nat Med 24:1716
Kononenko I (2001) Machine learning for medical diagnosis: history, state of the art and perspective. Artif Intell Med 23:89–109
Lasko TA, Joshua CD, Mia AL (2013) Computational phenotype discovery using unsupervised feature learning over noisy, sparse, and irregular clinical data, PloS one 8
Lavrač N (1999) Machine learning for data mining in medicine. In: Joint European conference on artificial intelligence in medicine and medical decision making, Springer, 47–62
LeCun Y, Yoshua B, Geoffrey H (2015) Deep learning. Nature 521:436–444
Lee H, Shin S-Y, Seo M, Nam G-B, Joo S (2016) Prediction of ventricular tachycardia one hour before occurrence using artificial neural networks. Sci Rep 6:32390
Lee H, Shin S-Y, Seo M, Nam G-B, Joo S (2016) Prediction of ventricular tachycardia one hour before occurrence using artificial neural networks. Sci Rep 6:1–7
Lee J, Mark RG (2010a) A hypotensive episode predictor for intensive care based on heart rate and blood pressure time series. In: 2010 computing in cardiology, IEEE, 81–84
Lee J, Mark RG (2010) An investigation of patterns in hemodynamic data indicative of impending hypotension in intensive care. Biomed Eng Online 9:62
Legates DR, Mccabe GJ Jr (1999) Evaluating the use of “goodness-of-fit” measures in hydrologic and hydroclimatic model validation. Water Resour Res 35:233–241
Li Q, Clifford GD (2012) Signal quality and data fusion for false alarm reduction in the intensive care unit. J Electrocardiol 45:596–603
Liaw SY, Scherpbier A, Klainin-Yobas P, Rethans J-J (2011) A review of educational strategies to improve nurses’ roles in recognizing and responding to deteriorating patients. Int Nurs Rev 58:296–303
Liu V, Kipnis P, Rizk NW, Escobar GJ (2012) Adverse outcomes associated with delayed intensive care unit transfers in an integrated healthcare system. J Hosp Med 7:224–230
Liu Z, Zuren F, Liangjun K (2015) Fireworks algorithm for the multi-satellite control resource scheduling problem. In: 2015 IEEE congress on evolutionary computation (CEC), IEEE, 1280–86
Manning T, Sleator RD, Walsh P (2014) Biologically inspired intelligent decision making: a commentary on the use of artificial neural networks in bioinformatics. Bioengineered 5:80–95
Mao Y, Wenlin C, Yixin C, Chenyang L, Marin K, Thomas B (2012) An integrated data mining approach to real-time clinical monitoring and deterioration warning. In: Proceedings of the 18th ACM SIGKDD international conference on knowledge discovery and data mining, 1140–48
Mardini L, Lipes J, Jayaraman D (2012) Adverse outcomes associated with delayed intensive care consultation in medical and surgical inpatients. J Crit Care 27:688–693
Masud MM, Al Harahsheh AR (2016) Mortality prediction of ICU patients using lab test data by feature vector compaction and classification. In: 2016 IEEE international conference on big data (big data), IEEE, 3404–11
Mochizuki K, Shintani R, Mori K, Sato T, Sakaguchi O, Takeshige K, Nitta K, Imamura H (2017) Importance of respiratory rate for the prediction of clinical deterioration after emergency department discharge: a single-center, case–control study. Acute Med Surg 4:172–178
Mokart D, Lambert J, Schnell D, Fouché L, Rabbat A, Kouatchet A, Lemiale V, Vincent F, Lengliné E, Bruneel F (2013) Delayed intensive care unit admission is associated with increased mortality in patients with cancer with acute respiratory failure. Leuk Lymphoma 54:1724–1729
Moody GB, Lehman L-WH (2009) Predicting acute hypotensive episodes: the 10th annual physionet/computers in cardiology challenge. In: 2009 36th annual computers in cardiology conference (CinC), IEEE, 541-44
Morgan, RJMWF, Lloyd-Williams F, Wright MM, Morgan-Warren RJ (1997) An early warning scoring system for detecting developing critical illness
Morris PE, Berry MJ, Clark Files D, Clifton Thompson J, Hauser J, Flores L, Dhar S, Chmelo E, Lovato J, Douglas L, Case. (2016) Standardized rehabilitation and hospital length of stay among patients with acute respiratory failure: a randomized clinical trial. JAMA 315:2694–2702
Newman S (2017) Do not disturb: vital sign monitoring as a predictor of clinical deterioration in monitored patients, Kentucky Nurs, 65
Nicolson A, Paliwal KK (2019) Deep learning for minimum mean-square error approaches to speech enhancement. Speech Commun 111:44–55
Oellrich A, Collier N, Groza T, Rebholz-Schuhmann D, Shah N, Bodenreider O, Boland MR, Georgiev I, Liu H, Livingston K (2016) The digital revolution in phenotyping. Brief Bioinform 17:819–830
Ong ME, Hock CH, Ng L, Goh K, Liu N, Koh ZX, Shahidah N, Zhang TT, Fook-Chong S, Lin Z (2012) Prediction of cardiac arrest in critically ill patients presenting to the emergency department using a machine learning score incorporating heart rate variability compared with the modified early warning score. Crit Care 16:R108
Ordoñez P, Schwarz N, Figueroa-Jiménez A, Garcia-Lebron LA, Roche-Lima A (2016) Learning stochastic finite-state transducer to predict individual patient outcomes. Health Technol 6:239–245
Panday RSN, Minderhoud TC, Alam N, Nanayakkara PWB (2017) Prognostic value of early warning scores in the emergency department (ED) and acute medical unit (AMU): a narrative review. Eur J Intern Med 45:20–31
Paradiso R (2003) Wearable health care system for vital signs monitoring. In: 4th international IEEE EMBS special topic conference on information technology applications in biomedicine, IEEE, 283–86.
Pawitan Y (2001) In all likelihood: statistical modelling and inference using likelihood. Oxford University Press
Pirracchio R (2016) Mortality prediction in the icu based on mimic-ii results from the super icu learner algorithm (sicula) project. Secondary analysis of electronic health records. Springer
Plate JDJ, Peelen LM, Leenen LPH, Hietbrink F (2018) Validation of the VitalPAC early warning score at the intermediate care unit. World J Critical Care Med 7:39
Polley EC, van der Laan MJ (2010) Super learner in prediction. Springer
Prytherch DR, Smith GB, Schmidt P, Featherstone PI, Stewart K, Knight D, Higgins B (2006) Calculating early warning scores—a classroom comparison of pen and paper and hand-held computer methods. Resuscitation 70:173–178
Purushotham S, Chuizheng M, Zhengping C, Yan L (2017) Benchmark of deep learning models on large healthcare mimic datasets, arXiv preprint arXiv:1710.08531
Qi J, Jun Du, Siniscalchi SM, Ma X, Lee C-H (2020) On mean absolute error for deep neural network based vector-to-vector regression. IEEE Signal Process Lett 27:1485–1489
Quinten VM, van Meurs M, Olgers TJ, Vonk JM, Ligtenberg JJM, ter Maaten JC (2018) Repeated vital sign measurements in the emergency department predict patient deterioration within 72 hours: a prospective observational study. Scand J Trauma Resusc Emerg Med 26:57
Rafiq M, George K, Pamela M, Jonas S, Carl S, and Christian G (2018) Deep learning architectures for vector representations of patients and exploring predictors of 30-day hospital readmissions in patients with multiple chronic conditions. In: International workshop on artificial intelligence in health, Springer, 228–44
Rajkomar A, Oren E, Chen K, Dai AM, Hajaj N, Hardt M, Liu PJ, Liu X, Marcus J, Sun M (2018) Scalable and accurate deep learning with electronic health records. NPJ Digit Med 1:18
Ren J (2012) ANN vs. SVM: Which one performs better in classification of MCCs in mammogram imaging. Knowl-Based Syst 26:144–153
Reyes-García J, Galeana-Zapién H, Galaviz-Mosqueda A, Torres-Huitzil C (2018) Evaluation of the impact of data uncertainty on the prediction of physiological patient deterioration. IEEE Access 6:38595–38606
Rothman MJ, Rothman SI, Joseph Beals IV (2013) Development and validation of a continuous measure of patient condition using the electronic medical record. J Biomed Inform 46:837–848
Saeed M, Christine L, Greg R, Roger GM (2002) MIMIC II: a massive temporal ICU patient database to support research in intelligent patient monitoring. In: Computers in cardiology, IEEE, 641–44
Saeed M, Villarroel M, Reisner AT, Clifford G, Lehman L-W, Moody G, Heldt T, Kyaw TH, Moody B, Mark RG (2011) Multiparameter intelligent monitoring in intensive care II (MIMIC-II): a public-access intensive care unit database. Crit Care Med 39:952
Saito T, Rehmsmeier M (2015) The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE 10:e0118432
Scalzo F, Liebeskind D, Xiao Hu (2012) Reducing false intracranial pressure alarms using morphological waveform features. IEEE Trans Biomed Eng 60:235–239
Schmid F, Goepfert MS, Reuter DA (2013) Patient monitoring alarms in the ICU and in the operating room. Crit Care 17:216
Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, Bellomo R, Bernard GR, Chiche J-D, Coopersmith CM (2016) The third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA 315:801–810
Smith GB, Prytherch DR, Meredith P, Schmidt PE, Featherstone PI (2013) The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation 84:465–470
Smith GB, Prytherch DR, Schmidt PE, Featherstone PI (2008) Review and performance evaluation of aggregate weighted ‘track and trigger’systems. Resuscitation 77:170–179
Spångfors M, Arvidsson L, Karlsson V, Samuelson K (2016) The national early warning score: translation, testing and prediction in a Swedish setting. Intensive Crit Care Nurs 37:62–67
Stanzani M, Lewis RE (2018) Development and applications of prognostic risk models in the management of invasive mold disease. J Fungi 4:141
Strzelczyk A, Ansorge S, Hapfelmeier J, Vijayveer Bonthapally M, Erder H, Rosenow F (2017) Costs, length of stay, and mortality of super-refractory status epilepticus: a population-based study from Germany. Epilepsia 58:1533–1541
Subbe CP, Kruger M, Rutherford P, Gemmel L (2001) Validation of a modified early warning score in medical admissions. QJM 94:521–526
Tang CHH, Middleton PM, Savkin AV, Chan GSH, Bishop S, Lovell NH (2010) Non-invasive classification of severe sepsis and systemic inflammatory response syndrome using a nonlinear support vector machine: a preliminary study. Physiol Meas 31:775
Taylor MM, Douglas-Creelman C (1967) PEST: efficient estimates on probability functions. J Acoust Soc Am 41:782–787
Taylor RA, Joseph RP, Arjun KV, Hani M, Edward RM, William F, Kennedy-Hall M (2016) Prediction of in-hospital mortality in emergency department patients with sepsis: a local big data–driven, machine learning approach. Acad Emerg Med 23:269–278
Tilly KF, Belton AB, McLachlan JFC (1995) Continuous monitoring of health status outcomes: experience with a diabetes education program. Diabet Educ 21:413–419
Tlegenov Y, Hong GS, Wen Feng Lu (2018) Nozzle condition monitoring in 3D printing. Robot Comput-Integr Manuf 54:45–55
Wang W, Yanmin L (2018) Analysis of the mean absolute error (MAE) and the root mean square error (RMSE) in assessing rounding model. In: IOP conference series materials science and engineering, 012049
Wellner B, Grand J, Canzone E, Coarr M, Brady PW, Simmons J, Kirkendall E, Dean N, Kleinman M, Sylvester P (2017) Predicting unplanned transfers to the intensive care unit: a machine learning approach leveraging diverse clinical elements. JMIR Med Inform 5:e45
Wickramasinghe N (2017) Deepr: a convolutional net for medical records
Wiley JF, Pace LA (2015) Multiple regression beginning R. Springer
Williams B, Alberti G, Ball C, Ball D, Binks R, Durham L (2012) Royal college of physicians, national early warning score (NEWS), standardising the assessment of acute-illness severity in the NHS, London
Young MP, Gooder VJ, Bride KM, James B, Fisher ES (2003) Inpatient transfers to the intensive care unit: delays are associated with increased mortality and morbidity. J Gen Intern Med 18:77–83
Zhai H, Brady P, Li Qi, Lingren T, Ni Y, Wheeler DS, Solti I (2014) Developing and evaluating a machine learning based algorithm to predict the need of pediatric intensive care unit transfer for newly hospitalized children. Resuscitation 85:1065–1071
Zheng Y, Qi L, Enhong C, Yong G, Leon-Zhao J (2016) Exploiting multi-channels deep convolutional neural networks for multivariate time series classification. Front Comput Sci 10:96–112
Acknowledgement
This work was supported by the Ministry of Higher Education under Prototype Research Grant Scheme (PRGS/1/2019/TK04/UTM/02/12), and in part by the UTM International Doctoral Fellowship.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Al-Shwaheen, T.I., Moghbel, M., Hau, Y.W. et al. Use of learning approaches to predict clinical deterioration in patients based on various variables: a review of the literature. Artif Intell Rev 55, 1055–1084 (2022). https://doi.org/10.1007/s10462-021-09982-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10462-021-09982-2