Process Monitoring in the Intensive Care Unit: Assessing Patient Mobility Through Activity Analysis with a Non-Invasive Mobility Sensor

  • Austin ReiterEmail author
  • Andy Ma
  • Nishi Rawat
  • Christine Shrock
  • Suchi Saria
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9900)


Throughout a patient’s stay in the Intensive Care Unit (ICU), accurate measurement of patient mobility, as part of routine care, is helpful in understanding the harmful effects of bedrest [1]. However, mobility is typically measured through observation by a trained and dedicated observer, which is extremely limiting. In this work, we present a video-based automated mobility measurement system called NIMS: Non-Invasive Mobility Sensor. Our main contributions are: (1) a novel multi-person tracking methodology designed for complex environments with occlusion and pose variations, and (2) an application of human-activity attributes in a clinical setting. We demonstrate NIMS on data collected from an active patient room in an adult ICU and show a high inter-rater reliability using a weighted Kappa statistic of 0.86 for automatic prediction of the highest level of patient mobility as compared to clinical experts.


Activity recognition Tracking Patient safety 


  1. 1.
    Brower, R.: Consequences of bed rest. Crit. Care Med. 37(10), S422–S428 (2009)CrossRefGoogle Scholar
  2. 2.
    Corchado, J., Bajo, J., De Paz, Y., Tapia, D.: Intelligent environment for monitoring Alzheimer patients, agent technology for health care. Decis. Support Syst. 44(2), 382–396 (2008)CrossRefGoogle Scholar
  3. 3.
    Hwang, J., Kang, J., Jang, Y., Kim, H.: Development of novel algorithm and real-time monitoring ambulatory system using bluetooth module for fall detection in the elderly. In: IEEE EMBS (2004)Google Scholar
  4. 4.
    Smith, M., Saunders, R., Stuckhardt, K., McGinnis, J.: Best Care at Lower Cost: the Path to Continuously Learning Health Care in America. National Academies Press, Washington, DC (2013)Google Scholar
  5. 5.
    Hashem, M., Nelliot, A., Needham, D.: Early mobilization and rehabilitation in the intensive care unit: moving back to the future. Respir. Care 61, 971–979 (2016)CrossRefGoogle Scholar
  6. 6.
    Berney, S., Rose, J., Bernhardt, J., Denehy, L.: Prospective observation of physical activity in critically ill patients who were intubated for more than 48 hours. J. Crit. Care 30(4), 658–663 (2015)CrossRefGoogle Scholar
  7. 7.
    Chakraborty, I., Elgammal, A., Burd, R.: Video based activity recognition in trauma resuscitation. In: International Conference on Automatic Face and Gesture Recognition (2013)Google Scholar
  8. 8.
    Lea, C., Facker, J., Hager, G., et al.: 3D sensing algorithms towards building an intelligent intensive care unit. In: AMIA Joint Summits Translational Science Proceedings (2013)Google Scholar
  9. 9.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE CVPR (2005)Google Scholar
  10. 10.
    Chen, X., Mottaghi, R., Liu, X., et al.: Detect what you can: detecting and representing objects using holistic models and body parts. In: IEEE CVPR (2014)Google Scholar
  11. 11.
    Felzenszwalb, P., Girshick, R., McAllester, D., Ramanan, D.: Object detection with discriminatively trained part-based models. PAMI 32(9), 1627–1645 (2010)CrossRefGoogle Scholar
  12. 12.
    Verceles, A., Hager, E.: Use of accelerometry to monitor physical activity in critically ill subjects: a systematic review. Respir. Care 60(9), 1330–1336 (2015)CrossRefGoogle Scholar
  13. 13.
    Babenko, D., Yang, M., Belongie, S.: Robust object tracking with online multiple instance learning. PAMI 33(8), 1619–1632 (2011)CrossRefGoogle Scholar
  14. 14.
    Lu, Y., Wu, T., Zhu, S.: Online object tracking, learning and parsing with and-or graphs. In: IEEE CVPR (2014)Google Scholar
  15. 15.
    Choi, W., Pantofaru, C., Savarese, S.: A general framework for tracking multiple people from a moving camera. PAMI 35(7), 1577–1591 (2013)CrossRefGoogle Scholar
  16. 16.
    Milan, A., Roth, S., Schindler, K.: Continuous energy minimization for multi-target tracking. TPAMI 36(1), 58–72 (2014)CrossRefGoogle Scholar
  17. 17.
    Wang, H., Schmid, C.: Action recognition with improved trajectories. In: IEEE ICCV (2013)Google Scholar
  18. 18.
    Karpathy, A., Toderici, G., Shetty, S., et al.: Large-scale video classification with convolutional neural networks. In: IEEE CVPR (2014)Google Scholar
  19. 19.
    Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: NIPS (2014)Google Scholar
  20. 20.
    Wu, Z., Wang, X., Jiang, Y., Ye, H., Xue, X.: Modeling spatial-temporal clues in a hybrid deep learning framework for video classification. In: ACMMM (2015)Google Scholar
  21. 21.
    Liu, J., Kuipers, B., Savarese, S.: Recognizing human actions by attributes. In: IEEE CVPR (2011)Google Scholar
  22. 22.
    Ma, A.J., Yuen, P.C., Saria, S.: Deformable distributed multiple detector fusion for multi-person tracking (2015). arXiv:1512.05990 [cs.CV]
  23. 23.
    Hodgson, C., Needham, D., Haines, K., et al.: Feasibility and inter-rater reliability of the ICU mobility scale. Heart Lung 43(1), 19–24 (2014)CrossRefGoogle Scholar
  24. 24.
    Girshick, R.: Fast R-CNN (2015). arXiv:1504.08083
  25. 25.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS (2012)Google Scholar
  26. 26.
    Keni, B., Rainer, S.: Evaluating multiple object tracking performance: the CLEAR MOT metrics. EURASIP J. Image Video Proces. 2008, 1–10 (2008)Google Scholar
  27. 27.
    Spinello, L., Arras, K.O.: People detection in RGB-D data. In: IROS (2011)Google Scholar
  28. 28.
    McHugh, M.: Interrater reliability: the Kappa statistic. Biochemia Med. 22(3), 276–282 (2012)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Austin Reiter
    • 1
    Email author
  • Andy Ma
    • 1
  • Nishi Rawat
    • 2
  • Christine Shrock
    • 2
  • Suchi Saria
    • 1
  1. 1.The Johns Hopkins UniversityBaltimoreUSA
  2. 2.Johns Hopkins Medical InstitutionsBaltimoreUSA

Personalised recommendations