Skip to main content
SpringerLink
Log in

Three-Stream Convolutional Neural Network for Human Fall Detection

  • Chapter
  • First Online:
Deep Learning Applications, Volume 2

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 1232))

Abstract

Lower child mortality rates, advances in medicine, and cultural changes have increased life expectancy to above 60-years old in developed countries. Some countries expect that, by 2030, 20% of their population will be over 65 years old. The quality of life at this advanced age is highly dictated by the individual’s health, which will determine whether the elderly can engage in important activities to their well-being, independence, and personal satisfaction. Old age is accompanied by health problems caused by biological limitations and muscle weakness. This weakening facilitates the occurrence of falls, which are responsible for the deaths of approximately 646,000 people worldwide and, even when a minor fall occurs, it can still cause fractures, break bones, or damage soft tissues, which will not heal completely. Injuries and damages of this nature, in turn, will consume the self-confidence of the individual, diminishing their independence. In this work, we propose a method capable of detecting human falls in video sequences using multi-channel convolutional neural networks (CNN). Our method makes use of a 3D CNN fed with features previously extracted from each frame to generate a vector for each channels. Then, the vectors are concatenated, and a support vector machine (SVM) is applied to classify the vectors and indicate whether or not there was a fall. We experiment with four types of features, namely: (i) optical flow, (ii) visual rhythm, (iii) pose estimation, and (iv) saliency map. The benchmarks used (UR Fall Detection Dataset (URFD) [33] and (ii) Fall Detection Dataset (FDD) [12]) are publicly available and our results are compared to those in the literature. The metrics selected for evaluation are balanced accuracy, accuracy, sensitivity, and specificity. Our results are competitive with those obtained by the state of the art on both URFD and FDD datasets. To the authors’ knowledge, we are the first to perform cross-tests between the datasets in question and to report results for the balanced accuracy metric. The proposed method is able to detect falls in the selected benchmarks. Fall detection, as well as activity classification in videos, is strongly related to the network’s ability to interpret temporal information and, as expected, optical flow is the most relevant feature for detecting falls.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, TensorFlow: large-scale machine learning on heterogeneous systems (2015). https://www.tensorflow.org

  2. A. Abobakr, M. Hossny, S. Nahavandi, A skeleton-free fall detection system from depth images using random decision forest. IEEE Syst. J. 12(3), 2994–3005 (2017)

    Article  Google Scholar 

  3. D.T. Anderson, J.M. Keller, M. Skubic, X. Chen, Z. He, Recognizing falls from silhouettes, in International Conference of the IEEE Engineering in Medicine and Biology Society (2006), pp. 6388–6391

    Google Scholar 

  4. L. Anishchenko, Machine learning in video surveillance for fall detection, in Ural Symposium on Biomedical Engineering, Radioelectronics and Information Technology (IEEE, 2018), pp. 99–102

    Google Scholar 

  5. S. Bhandari, N. Babar, P. Gupta, N. Shah, S. Pujari, A novel approach for fall detection in home environment, in IEEE 6th Global Conference on Consumer Electronics (IEEE, 2017), pp. 1–5

    Google Scholar 

  6. G. Bradski, The openCV library. Dobb’s J. Softw. Tools 120, 122–125 (2000)

    Google Scholar 

  7. Z. Cao, T. Simon, S.E. Wei, Y. Sheikh, Realtime multi-person 2D pose estimation using part affinity fields, in IEEE Conference on Computer Vision and Pattern Recognition (2017), pp. 7291–7299

    Google Scholar 

  8. S. Carneiro, G. Silva, G. Leite, R. Moreno, S. Guimaraes, H. Pedrini, Deep convolutional multi-stream network detection system applied to fall identification in video sequences, in 15th International Conference on Machine Learning and Data Mining (2019a), pp. 681–695

    Google Scholar 

  9. S. Carneiro, G. Silva, G. Leite, R. Moreno, S. Guimaraes, H. Pedrini, Multi-stream deep convolutional network using high-level features applied to fall detection in video sequences, in 26th International Conference on Systems, Signals and Image Processing (2019b), pp. 293–298

    Google Scholar 

  10. J. Carreira, A. Zisserman, Quo vadis, action recognition? a new model and the kinetics dataset, in Conference on Computer Vision and Pattern Recognition (IEEE, 2017), pp. 6299–6308

    Google Scholar 

  11. I. Charfi, J. Miteran, J. Dubois, M. Atri, R. Tourki, Definition and performance evaluation of a robust svm based fall detection solution, in International Conference on Signal Image Technology and Internet Based Systems, vol. 12 (2012), pp. 218–224

    Google Scholar 

  12. I. Charfi, J. Miteran, J. Dubois, M. Atri, R. Tourki, Optimized spatio-temporal descriptors for real-time fall detection: comparison of support vector machine and adaboost-based classification. J. Electron. Imaging 22(4), 041106 (2013)

    Google Scholar 

  13. F. Chollet, Keras (2015). https://keras.io

  14. J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, L. Fei-Fei, Imagenet: a large–scale hierarchical image database, in IEEE Conference on Computer Vision and Pattern Recognition (2009), pp. 248–255

    Google Scholar 

  15. L. Deng, D. Yu, Deep learning: methods and applications. Found. Trends Signal Process. 7(3–4), 197–387 (2014)

    Google Scholar 

  16. A. Edgcomb, F. Vahid, Automated fall detection on privacy-enhanced video, in Annual International Conference of the IEEE Engineering in Medicine and Biology Society (2012), pp. 252–255

    Google Scholar 

  17. K. Fan, P. Wang, S. Zhuang, Human fall detection using slow feature analysis. Multimed. Tools Appl. 78(7), 9101–9128 (2018a)

    Article  Google Scholar 

  18. Y. Fan, G. Wen, D. Li, S. Qiu, M.D. Levine, Early event detection based on dynamic images of surveillance videos. J. Vis. Commun. Image Represent. 51, 70–75 (2018b)

    Article  Google Scholar 

  19. G. Farnebäck, Two–frame motion estimation based on polynomial expansion, in Scandinavian Conference on Image Analysis (2003), pp. 363–370

    Google Scholar 

  20. S. Gasparrini, E. Cippitelli, E. Gambi, S. Spinsante, J. Wåhslén, I. Orhan, T. Lindh, Proposal and experimental evaluation of fall detection solution based on wearable and depth data fusion, in International Conference on ICT Innovations (Springer, 2015), pp. 99–108

    Google Scholar 

  21. M.A. Goodale, A.D. Milner, Separate visual pathways for perception and action. Trends Neurosci. 15(1), 20–25 (1992)

    Article  Google Scholar 

  22. I. Goodfellow, Y. Bengio, A. Courville, Y. Bengio, Deep Learning (MIT Press, 2016)

    Google Scholar 

  23. F. Harrou, N. Zerrouki, Y. Sun, A. Houacine, Vision-based fall detection system for improving safety of elderly people. IEEE Instrum. & Meas. Mag. 20(6), 49–55 (2017)

    Article  Google Scholar 

  24. K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 770–778

    Google Scholar 

  25. D.L. Heymann, T. Prentice, L.T. Reinders, The World Health Report: a Safer Future: global Public Health Security in the 21st Century (World Health Organization, 2007)

    Google Scholar 

  26. Z. Huang, Y. Liu, Y. Fang, B.K. Horn, Video-based fall detection for seniors with human pose estimation, in 4th International Conference on Universal Village (IEEE, 2018), pp. 1–4

    Google Scholar 

  27. E. Jones, T. Oliphant, P. Peterson, SciPy: open source scientific tools for python (2001). http://www.scipy.org

  28. O.O. Khin, Q.M. Ta, C.C. Cheah, Development of a wireless sensor network for human fall detection, in International Conference on Real-Time Computing and Robotics (IEEE, 2017), pp. 273–278

    Google Scholar 

  29. Y. Kong, J. Huang, S. Huang, Z. Wei, S. Wang, Learning spatiotemporal representations for human fall detection in surveillance video. J. Vis. Commun. Image Represent. 59, 215–230 (2019)

    Article  Google Scholar 

  30. A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25, 1097–1105 (2012)

    Google Scholar 

  31. T. Kukharenko, V. Romanenko, Picking a human fall detection algorithm for wrist–worn electronic device, in IEEE First Ukraine Conference on Electrical and Computer Engineering (2017), pp. 275–277

    Google Scholar 

  32. V.S. Kumar, K.G. Acharya, B. Sandeep, T. Jayavignesh, A. Chaturvedi, Wearable sensor–based human fall detection wireless system, in Wireless Communication Networks and Internet of Things (Springer, 2018), pp. 217–234

    Google Scholar 

  33. B. Kwolek, M. Kepski, Human fall detection on embedded platform using depth maps and wireless accelerometer. Comput. Methods Programs Biomed. 117(3), 489–501 (2014)

    Article  Google Scholar 

  34. B. Kwolek, M. Kepski, Improving fall detection by the use of depth sensor and accelerometer. Neurocomputing 168, 637–645 (2015)

    Article  Google Scholar 

  35. Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  36. T. Lee, A. Mihailidis, An intelligent emergency response system: preliminary development and testing of automated fall Detection. J. Telemed. Telecare 11(4), 194–198 (2005)

    Article  Google Scholar 

  37. G. Leite, G. Silva, H. Pedrini, Fall detection in video sequences based on a three-stream convolutional neural network, in 18th IEEE International Conference on Machine Learning and Applications (ICMLA) (Boca Raton-FL, USA, 2019), pp. 191–195

    Google Scholar 

  38. G. Leite, G. Silva, H. Pedrini, Fall detection (2020). https://github.com/Lupins/fall_detection

  39. H. Li, K. Mueller, X. Chen, Beyond saliency: understanding convolutional neural networks from saliency prediction on layer-wise relevance propagation. Comput. Res. Repos. (2017a)

    Google Scholar 

  40. X. Li, T. Pang, W. Liu, T. Wang, Fall detection for elderly person care using convolutional neural networks, in 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (2017b), pp. 1–6

    Google Scholar 

  41. W.N. Lie, A.T. Le, G.H. Lin, Human fall-down event detection based on 2D skeletons and deep learning approach, in International Workshop on Advanced Image Technology (2018), pp. 1–4

    Google Scholar 

  42. B.S. Lin, J.S. Su, H. Chen, C.Y. Jan, A fall detection system based on human body silhouette, in 9th International Conference on Intelligent Information Hiding and Multimedia Signal Processing (IEEE, 2013), pp. 49–52

    Google Scholar 

  43. N. Lu, Y. Wu, L. Feng, J. Song, Deep learning for fall detection: 3D-CNN combined with LSTM on video kinematic data. IEEE J. Biomed. Health Inform. 23(1), 314–323 (2018)

    Article  Google Scholar 

  44. B.D. Lucas, T. Kanade, An iterative image registration technique with an application to stereo vision, in International Joint Conference on Artificial Inteligence (1981), pp. 121–130

    Google Scholar 

  45. F. Luna-Perejon, J. Civit-Masot, I. Amaya-Rodriguez, L. Duran-Lopez, J.P. Dominguez-Morales, A. Civit-Balcells, A. Linares-Barranco, An automated fall detection system using recurrent neural networks, in Conference on Artificial Intelligence in Medicine in Europe (Springer, 2019), pp. 36–41

    Google Scholar 

  46. M.M. Lusardi, S. Fritz, A. Middleton, L. Allison, M. Wingood, E. Phillips, Determining risk of falls in community dwelling older adults: a systematic review and meta-analysis using posttest probability. J. Geriatr. Phys. Ther. 40(1), 1–36 (2017)

    Article  Google Scholar 

  47. X. Ma, H. Wang, B. Xue, M. Zhou, B. Ji, Y. Li, Depth-based human fall detection via shape features and improved extreme learning machine. J. Biomed. Health Inform. 18(6), 1915–1922 (2014)

    Article  Google Scholar 

  48. L. Meng, B. Zhao, B. Chang, G. Huang, W. Sun, F. Tung, L. Sigal, Interpretable Spatio-Temporal Attention for Video Action Recognition (2018), pp. 1–10. arXiv preprint arXiv:181004511

  49. W. Min, H. Cui, H. Rao, Z. Li, L. Yao, Detection of human falls on furniture using scene analysis based on deep learning and activity Characteristics. IEEE Access 6, 9324–9335 (2018)

    Article  Google Scholar 

  50. M.N.H. Mohd, Y. Nizam, S. Suhaila, M.M.A. Jamil, An optimized low computational algorithm for human fall detection from depth images based on support vector machine classification, in IEEE International Conference on Signal and Image Processing Applications (2017), pp. 407–412

    Google Scholar 

  51. T.P. Moreira, D. Menotti, H. Pedrini, First-person action recognition through visual rhythm texture description, in International Conference on Acoustics (Speech and Signal Processing, IEEE, 2017), pp. 2627–2631

    Google Scholar 

  52. E.B. Nievas, O.D. Suarez, G.B. García, R. Sukthankar, Violence detection in video using computer vision techniques, in International Conference on Computer Analysis of Images and Patterns (Springer, 2011), pp. 332–339

    Google Scholar 

  53. Y. Nizam, M.N.H. Mohd, M.M.A. Jamil, Human fall detection from depth images using position and velocity of subject. Procedia Comput. Sci. 105, 131–137 (2017)

    Article  Google Scholar 

  54. A. Núñez-Marcos, G. Azkune, I. Arganda-Carreras, Vision-based fall detection with convolutional neural networks. Wirel. Commun. Mob. Comput. 2017, 1–16 (2017)

    Article  Google Scholar 

  55. T.E. Oliphant, Guide to NumPy, 2nd edn. (CreateSpace Independent Publishing Platform, USA, USA, 2015)

    Google Scholar 

  56. L. Panahi, V. Ghods, Human fall detection using machine vision techniques on RGB-D images. Biomed. Signal Process. Control 44, 146–153 (2018)

    Article  Google Scholar 

  57. P.S. Sase, S.H. Bhandari, Human fall detection using depth videos, in 5th International Conference on Signal Processing and Integrated Networks (IEEE, 2018), pp. 546–549

    Google Scholar 

  58. C. Schuldt, I. Laptev, B. Caputo, Recognizing human actions: a local SVM approach, in 17th International Conference on Pattern Recognition, vol. 3 (IEEE, 2004), pp 32–36

    Google Scholar 

  59. K. Sehairi, F. Chouireb, J. Meunier, Elderly fall detection system based on multiple shape features and motion analysis, in International Conference on Intelligent Systems and Computer Vision (IEEE, 2018), pp. 1–8

    Google Scholar 

  60. A. Shojaei-Hashemi, P. Nasiopoulos, J.J. Little, M.T. Pourazad, Video–based human fall detection in smart homes using deep learning, in IEEE International Symposium on Circuits and Systems (2018), pp. 1–5

    Google Scholar 

  61. K. Simonyan, A. Zisserman, Two-stream convolutional networks for action recognition in videos. Adv. Neural Inf. Process. Syst. 27, 568–576 (2014a)

    Google Scholar 

  62. K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition (2014b), pp. 1–14. arXiv, arXiv:14091556

  63. K. Simonyan, A. Vedaldi, A. Zisserman, Deep inside convolutional networks: visualising image classification models and saliency maps. Comput. Res. Repos. (2013)

    Google Scholar 

  64. D. Smilkov, N. Thorat, B. Kim, F. Viégas, M. Wattenberg, Smoothgrad: removing noise by adding noise (2017), pp. 1–10. arXiv preprint arXiv:170603825

  65. M. Sundararajan, A. Taly, Q. Yan, Axiomatic attribution for deep networks, in 34th International Conference on Machine Learning, vol. 70, pp. 3319–3328 (JMLR.org, 2017)

    Google Scholar 

  66. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, going deeper with convolutions, in IEEE Conference on Computer Vision and Pattern Recognition (2015), pp. 1–9

    Google Scholar 

  67. S.K. Tasoulis, G.I. Mallis, S.V. Georgakopoulos, A.G. Vrahatis, V.P. Plagianakos, I.G. Maglogiannis, Deep learning and change detection for fall recognition, in Engineering Applications of Neural Networks, ed. by J. Macintyre, L. Iliadis, I. Maglogiannis, C. Jayne (Springer International Publishing, Cham, 2019), pp. 262–273

    Google Scholar 

  68. The Joint Commission, Fall reduction program—definition of a fall (2001)

    Google Scholar 

  69. B.S. Torres, H. Pedrini, Detection of complex video events through visual rhythm. Vis. Comput. 34(2), 145–165 (2018)

    Article  Google Scholar 

  70. US Department of Veterans Affairs, Falls policy overview (2019). http://www.patientsafety.va.gov/docs/fallstoolkit14/05_falls_policy_overview_v5.docx

  71. F.B. Valio, H. Pedrini, N.J. Leite, Fast rotation-invariant video caption detection based on visual rhythm. in Iberoamerican Congress on Pattern Recognition (Springer, 2011), pp. 157–164

    Google Scholar 

  72. M. Vallejo, C.V. Isaza, J.D. Lopez, Artificial neural networks as an alternative to traditional fall detection methods, in 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (2013), pp. 1648–1651

    Google Scholar 

  73. G. Van Rossum, F.L. Jr Drake, Python reference manual. Tech. Rep. Report CS-R9525, Centrum voor Wiskunde en Informatica, Amsterdam (1995)

    Google Scholar 

  74. L. Wang, Y. Xiong, Z. Wang, Y. Qiao, Towards good practices for very deep two-stream convnets (2015), pp. 1–5. arXiv preprint arXiv:150702159

  75. M. Wani, F. Bhat, S. Afzal, A. Khan, Advances in Deep Learning (Springer, 2020)

    Google Scholar 

  76. World Health Organization, Global Health and Aging (2011)

    Google Scholar 

  77. World Health Organization, Fact sheet falls (2012)

    Google Scholar 

  78. World Health Organization, World Report on Ageing and Health (2015)

    Google Scholar 

  79. T. Xu, Y. Zhou, J. Zhu, New advances and challenges of fall detection systems: a survey. Appl. Sci. 8(3), 418 (2018)

    Google Scholar 

  80. M. Yu, S.M. Naqvi, J. Chambers, A robust fall detection system for the elderly in a smart room, in IEEE International Conference on Acoustics Speech and Signal Processing (2010), pp. 1666–1669

    Google Scholar 

  81. N. Zerrouki, A. Houacine, Combined curvelets and hidden Markov models for human fall detection. Multimed. Tools Appl. 77(5), 6405–6424 (2018)

    Article  Google Scholar 

  82. N. Zerrouki, F. Harrou, Y. Sun, A. Houacine, Vision-based human action classification using adaptive boosting algorithm. IEEE Sens. J. 18(12), 5115–5121 (2018)

    Article  Google Scholar 

  83. Z. Zhang, V. Athitsos, Fall detection by zhong zhang and vassilis athitsos (2020). http://vlm1.uta.edu/~zhangzhong/fall_detection/

  84. S. Zhao, W. Li, W. Niu, R. Gravina, G. Fortino, Recognition of human fall events based on single tri–axial gyroscope, in IEEE 15th International Conference on Networking, Sensing and Control (2018), pp. 1–6

    Google Scholar 

  85. F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, Q. He, A comprehensive survey on transfer learning (2019), pp. 1–27. arXiv preprint arXiv:191102685

  86. Y. Zigel, D. Litvak, I. Gannot, A method for automatic fall detection of elderly people using floor vibrations and sound-proof of concept on human mimicking doll falls. IEEE Trans. Biomed. Eng. 56(12), 2858–2867 (2009)

    Article  Google Scholar 

  87. Z. Zuo, B. Wei, F. Chao, Y. Qu, Y. Peng, L. Yang, Enhanced gradient-based local feature descriptors by saliency map for egocentric action recognition. Appl. Syst. Innov. 2(1), 1–14 (2019)

    Article  Google Scholar 

Download references

Acknowledgements

The authors are thankful to FAPESP (grant #2017/12646-3), CNPq (grant #309330/2018-7), and CAPES for their financial support, as well as Semantix Brasil for the infrastructure and support provided during the development of the present work.

Author information

Authors and Affiliations

  1. Institute of Computing, University of Campinas, Campinas, SP, Brazil

    Guilherme Vieira Leite, Gabriel Pellegrino da Silva & Helio Pedrini

Authors
  1. Guilherme Vieira Leite
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gabriel Pellegrino da Silva
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Helio Pedrini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Helio Pedrini .

Editor information

Editors and Affiliations

  1. Department of Computer Science, University of Kashmir, Srinagar, India

    M. Arif Wani

  2. Computer and Electrical Engineering, Florida Atlantic University, Boca Raton, FL, USA

    Taghi M. Khoshgoftaar

  3. Faculty of Engineering and Computing, Coventry University, Coventry, UK

    Vasile Palade

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Leite, G.V., da Silva, G.P., Pedrini, H. (2021). Three-Stream Convolutional Neural Network for Human Fall Detection. In: Wani, M.A., Khoshgoftaar, T.M., Palade, V. (eds) Deep Learning Applications, Volume 2. Advances in Intelligent Systems and Computing, vol 1232. Springer, Singapore. https://doi.org/10.1007/978-981-15-6759-9_3

Download citation

Publish with us

Policies and ethics

Search

Navigation