Skip to main content
SpringerLink
Log in

DeepFall: Non-Invasive Fall Detection with Deep Spatio-Temporal Convolutional Autoencoders

  • Research Article
  • Published:
Journal of Healthcare Informatics Research Aims and scope Submit manuscript

Abstract

Human falls rarely occur; however, detecting falls is very important from the health and safety perspective. Due to the rarity of falls, it is difficult to employ supervised classification techniques to detect them. Moreover, in these highly skewed situations, it is also difficult to extract domain-specific features to identify falls. In this paper, we present a novel framework, DeepFall, which formulates the fall detection problem as an anomaly detection problem. The DeepFall framework presents the novel use of deep spatio-temporal convolutional autoencoders to learn spatial and temporal features from normal activities using non-invasive sensing modalities. We also present a new anomaly scoring method that combines the reconstruction score of frames across a temporal window to detect unseen falls. We tested the DeepFall framework on three publicly available datasets collected through non-invasive sensing modalities, thermal camera and depth cameras, and show superior results in comparison with traditional autoencoder methods to identify unseen falls.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. The code for DSTCAE is available at https://github.com/JJN123/Fall-Detection.

References

  1. Baccouche M, Mamalet F, Wolf C, Garcia C, Baskurt A (2012) Spatio-temporal convolutional sparse autoencoder for sequence classification. In: BMVC. Citeseer

  2. Bertalmio M, Bertozzi AL, Sapiro G (2001) Navier-stokes, fluid dynamics, and image and video inpainting. In: Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001, vol 1, pp I–355–I–362, DOI https://doi.org/10.1109/CVPR.2001.990497

  3. Bogdan Kwolek MK (2014) Human fall detection on embedded platform using depth maps and wireless accelerometer. Comput Methods Programs Biomed 117:489–501

    Article  Google Scholar 

  4. CDC (2017) Enter of disease control and prevention – important facts about falls. https://www.cdc.gov/homeandrecreationalsafety/falls/adultfalls.html. [Online accessed 22-December-2017]

  5. Chalapathy R, Menon AK, Chawla S (2017) Robust, deep and inductive anomaly detection. In: European conference on machine learning. Skopje

  6. Chandola V, Banerjee A, Kumar V (2009) Anomaly detection: a survey. ACM Comput Surv 41(3):15:1–15:58. https://doi.org/10.1145/1541880.1541882

    Article  Google Scholar 

  7. Chollet F., et al. (2015) Keras: the python deep learning library https://github.com/fchollet/keras. Online accessed 20-January-2018

  8. Chong YS, Tay YH (2017) Abnormal event detection in videos using spatiotemporal autoencoder. In: International symposium on neural networks. Springer, pp 189–196

  9. Du Tran Lubomir Bourdev RFLTMP (2015) Learning spatiotemporal features with 3d convolutional networks. In: IEEE International conference on computer vision (ICCV). IEEE, DOI https://doi.org/10.1109/ICCV.2015.510

  10. Goldstein M, Uchida S (2016) A comparative evaluation of unsupervised anomaly detection algorithms for multivariate data. PloS one 11 4:e0152173

    Article  Google Scholar 

  11. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

  12. Gutoski M, Aquino NMR, Ribeiro M, Lazzaretti AE, Lopes HS (2017) Detection of video anomalies using convolutional autoencoders and one-class support vector machines. In: Proc. XIII Brazilian congress on computational intelligence

  13. Hasan M, Choi J, Neumann J, Roy-Chowdhury AK, Davis LS (2016) Learning temporal regularity in video sequences. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 733–742

  14. Itseez (2015) Open source computer vision library. https://github.com/itseez/opencv

  15. Ji S, Xu W, Yang M, Yu K (2013) 3d convolutional neural networks for human action recognition. IEEE Trans Pattern Anal Mach Intell 35(1):221–231

    Article  Google Scholar 

  16. Khan SS, Ahmad A (2018) Relationship between variants of one-class nearest neighbors and creating their accurate ensembles. IEEE Trans Knowl Data Eng 30 (9):1796–1809. https://doi.org/10.1109/TKDE.2018.2806975

    Article  Google Scholar 

  17. Khan SS, Karg ME, Kulić D, Hoey J (2017) Detecting falls with x-factor hidden Markov models. Appl Soft Comput 55:168–177

    Article  Google Scholar 

  18. Khan SS, Madden MG (2014) One-class classification: taxonomy of study and review of techniques. Knowl Eng Rev 29(3):345–374

    Article  Google Scholar 

  19. Khan SS, Taati B (2017) Detecting unseen falls from wearable devices using channel-wise ensemble of autoencoders. Expert Systems with Applications

  20. Ma X, Wang H, Xue B, Zhou M, Ji B, Li Y (2014) Depth-based human fall detection via shape features and improved extreme learning machine. IEEE J Biomed Health Inform 18(6):1915–1922. https://doi.org/10.1109/JBHI.2014.2304357

    Article  Google Scholar 

  21. Masci J, Meier U, Cireşan D, Schmidhuber J (2011) Stacked convolutional auto-encoders for hierarchical feature extraction. In: International conference on artificial neural networks. Springer, pp 52–59

  22. Mercuri M, Garripoli C, Karsmakers P, Soh PJ, Vandenbosch GA, Pace C, Leroux P, Schreurs D (2016) Healthcare system for non-invasive fall detection in indoor environment. In: Applications in electronics pervading industry, environment and society. Springer, pp 145–152

  23. Munawar A, Vinayavekhin P, De Magistris G (2017) Spatio-temporal anomaly detection for industrial robots through prediction in unsupervised feature space. In: 2017 IEEE Winter conference on applications of computer vision (WACV). IEEE, pp 1017–1025

  24. Nathan Silberman Derek Hoiem PK, Fergus R (2012) Indoor segmentation and support inference from rgbd images. In: ECCV

  25. Nogas J, Khan SS, Mihailidis A (2018) Fall detection from thermal camera using convolutional lstm autoencoder. In: Proceedings of the 2nd workshop on aging, rehabilitation and independent assisted living. IJCAI Workshop

  26. Penttilä J (2017) A method for anomaly detection in hyperspectral images, using deep convolutional autoencoders. Master’s thesis. University of Jyväskylä, Finland

    Google Scholar 

  27. Ribeiro M, Lazzaretti AE, Lopes HS (2017) A study of deep convolutional auto-encoders for anomaly detection in videos. Pattern Recognition Letters

  28. Rubenstein LZ, Robbins AS, Josephson KR, Schulman BL, Osterweil D (1990) The value of assessing falls in an elderly population: a randomized clinical trial. Ann Intern Med 113(4):308–316

    Article  Google Scholar 

  29. Sabokrou M, Fayyaz M, Fathy M, Klette R (2017) Deep-cascade: cascading 3d deep neural networks for fast anomaly detection and localization in crowded scenes. IEEE Trans Image Process 26(4):1992–2004

    Article  MathSciNet  Google Scholar 

  30. Samek W, Wiegand T, Müller KR (2017) Explainable artificial intelligence: understanding, visualizing and interpreting deep learning models. arXiv:https://arxiv.org/abs/1708.08296

  31. Skubic M, Harris BH, Stone E, Ho K, Su BY, Rantz M (2016) Testing non-wearable fall detection methods in the homes of older adults. In: 2016 IEEE 38th annual international conference of the engineering in medicine and biology society (EMBC). IEEE, pp 557–560

  32. Tran HT, Hogg D (2017) Anomaly detection using a convolutional winner-take-all autoencoder. In: Proceedings of the British machine vision conference 2017. Leeds

  33. Vadivelu S, Ganesan S, Murthy OR, Dhall A (2016) Thermal imaging based elderly fall detection. In: ACCV workshop. Springer, pp 541–553

  34. Viacheslav V, Alexander F, Vladimir M, Svetlana T, Oksana L (2014) Kinect depth map restoration using modified exemplar-based inpainting. In: 2014 12th International conference on signal processing (ICSP). IEEE, pp 1175–1179

  35. Xu D, Yan Y, Ricci E, Sebe N (2017) Detecting anomalous events in videos by learning deep representations of appearance and motion. Comput Vis Image Underst 156:117–127

    Article  Google Scholar 

  36. Yusif S, Soar J, Hafeez-Baig A (2016) Older people, assistive technologies, and the barriers to adoption: a systematic review. Int J Med Inform 94:112–116

    Article  Google Scholar 

  37. Zhao Y, Deng B, Shen C, Liu Y, Lu H, Hua XS (2017) Spatio-temporal autoencoder for video anomaly detection. In: Proceedings of the 2017 ACM on multimedia conference, MM ’17. ACM, New York

Download references

Author information

Authors and Affiliations

  1. University of Toronto, Toronto, Canada

    Jacob Nogas & Shehroz S. Khan

  2. Toronto Rehabilitation Institute, University Health Network, Toronto, Canada

    Alex Mihailidis

Authors
  1. Jacob Nogas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shehroz S. Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alex Mihailidis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jacob Nogas.

Ethics declarations

Conflict of interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nogas, J., Khan, S.S. & Mihailidis, A. DeepFall: Non-Invasive Fall Detection with Deep Spatio-Temporal Convolutional Autoencoders. J Healthc Inform Res 4, 50–70 (2020). https://doi.org/10.1007/s41666-019-00061-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41666-019-00061-4

Keywords

Search

Navigation