Skip to main content
SpringerLink
Log in

Vision-based techniques for fall detection in 360∘ videos using deep learning: Dataset and baseline results

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Alarming cases of falls in the elderly have triggered the rise of robust and cost-efficient systems for automated fall detection in humans. Although several potential solutions exist, they still have not achieved the desired level of robustness and acceptability. Lately, the proliferation of low-cost cameras coupled with deep learning techniques has transformed vision-based methods for fall detection. Motivated by this, in this paper, we present an alternate low-cost and efficient system for fall detection in 360∘ videos using deep learning. Towards this, we first built a well-balanced video dataset named Fall360. The Fall360 dataset contains video clips of several falls and non-fall actions, captured by a 360∘ camera mounted on the ceiling in a home-like environment. Secondly, we examined the performance of deep learning techniques that consist of several variants of hybrid CNN & LSTM, hybrid CNN & ConvLSTM, and 3D CNNs to test the effectiveness of the dataset in the fall detection task. Thirdly, to assess the performance of these techniques, we conducted an ablation study on a recently introduced multi-camera UP-Fall dataset. The deep learning models attained substantial improvement in recognition accuracy on both the fall datasets and have set the new state-of-the-art performance. Overall, our designed fall detection system using 360∘ videos, in addition to providing a better perspective, bestows a more suitable and low-cost alternative for the existing multi-camera-based fall detection systems. To encourage more study, we will make our in-house Fall360 dataset publicly available to the research community.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29
Fig. 30

Similar content being viewed by others

References

  1. Ajerla D, Mahfuz S, Zulkernine F (2019) A real-time patient monitoring framework for fall detection. Wirel Commun Mob Comput

  2. Alhimale L, Zedan H, Al-Bayatti A (2014) The implementation of an intelligent and video-based fall detection system using a neural network. Appl Soft Comput 18:59–69

    Article  Google Scholar 

  3. Aziz O, Klenk J, Schwickert L, Chiari L, Becker C, Park EJ, Mori G, Robinovitch SN (2017) Validation of accuracy of svm-based fall detection system using real-world fall and non-fall datasets. PLoS one 12(7):e0180318

    Article  Google Scholar 

  4. Bajones M, Fischinger D, Weiss A, Wolf D, Vincze M, de la Puente P, Körtner T, Weninger M, Papoutsakis K, Michel D et al (2018) Hobbit: Providing fall detection and prevention for the elderly in the real world. Journal of Robotics, 2018

  5. Boudouane I, Makhlouf A, Harkat MA, Hammouche MZ, Saadia N, Cherif AR (2019) Fall detection system with portable camera. Journal of Ambient Intelligence and Humanized Computing, 1–13

  6. Carreira J, Noland E, Hillier C, Zisserman A (2019) A short note on the kinetics-700 human action dataset. arXiv:190706987

  7. Casilari E, Oviedo-Jiménez MA (2015) Automatic fall detection system based on the combined use of a smartphone and a smartwatch. PloS One 10 (11):e0140929

    Article  Google Scholar 

  8. Chua JL, Chang YC, Lim WK (2015) A simple vision-based fall detection technique for indoor video surveillance. SIViP 9(3):623–633

    Article  Google Scholar 

  9. De Miguel K, Brunete A, Hernando M, Gambao E (2017) Home camera-based fall detection system for the elderly. Sensors 17(12):2864

    Article  Google Scholar 

  10. de la Concepción M A ́A ́, Morillo LMS, García JAÁ, González-Abril L (2017) Mobile activity recognition and fall detection system for elderly people using ameva algorithm. Pervasive and Mobile Computing 34:3–13

    Article  Google Scholar 

  11. Delgado-Escaño R, Castro FM, Cózar J R, Marín-jiménez MJ, Guil N, Casilari E (2020) A cross-dataset deep learning-based classifier for people fall detection and identification. Comput Methods Programs Biomed 184:105265

    Article  Google Scholar 

  12. de Quadros T, Lazzaretti AE, Schneider FK (2018) A movement decomposition and machine learning-based fall detection system using wrist wearable device. IEEE Sensors J 18(12):5082–5089

    Article  Google Scholar 

  13. Diraco G, Leone A, Siciliano P (2017) A radar-based smart sensor for unobtrusive elderly monitoring in ambient assisted living applications. Biosensors 7 (4):55

    Article  Google Scholar 

  14. Divya V, Leena RS (2020) Docker based intelligent fall detection using edge-fog cloud infrastructure. IEEE Internet of Things Journal

  15. Espinosa R, Ponce H, Gutiérrez S, Martínez-villaseñor L, Brieva J, Moya-Albor E (2019) A vision-based approach for fall detection using multiple cameras and convolutional neural networks: a case study using the up-fall detection dataset. Comput Biol Med 115:103520

    Article  Google Scholar 

  16. Feng Q, Gao C, Wang L, Zhao Y, Song T, Li Q (2020) Spatio-temporal fall event detection in complex scenes using attention guided lstm. Pattern Recogn Lett 130:242–249

    Article  Google Scholar 

  17. Geertsema EE, Visser GH, Viergever MA, Kalitzin SN (2019) Automated remote fall detection using impact features from video and audio. J Biomechan 88:25–32

    Article  Google Scholar 

  18. Gibson RM, Amira A, Ramzan N, Casaseca-de-la Higuera P, Pervez Z (2016) Multiple comparator classifier framework for accelerometer-based fall detection and diagnostic. Appl Soft Comput 39:94– 103

    Article  Google Scholar 

  19. Gibson RM, Amira A, Ramzan N, Casaseca-de-la Higuera P, Pervez Z (2017) Matching pursuit-based compressive sensing in a wearable biomedical accelerometer fall diagnosis device. Biomed Signal Process Control 33:96–108

    Article  Google Scholar 

  20. Gonzalez-Abril L, Cuberos FJ, Velasco F, Ortega JA (2009) Ameva: an autonomous discretization algorithm. Expert Syst Appl 36(3):5327–5332

    Article  Google Scholar 

  21. Greff K, Srivastava RK, Koutník J, Steunebrink BR, Schmidhuber J (2016) Lstm: A search space odyssey. IEEE Transactions on Neural Networks and Learning Systems 28(10):2222–2232

    Article  MathSciNet  Google Scholar 

  22. Guzmán JM, Pawliczko A, Beales S (2018) Ageing in the twenty-first century, A celebration and a challenge

  23. Hara K, Kataoka H, Satoh Y (2017) Learning spatio-temporal features with 3d residual networks for action recognition. In: Proceedings of the IEEE international conference on computer vision workshops, pp 3154–3160

  24. Hara K, Kataoka H, Satoh Y (2018) Towards good practice for action recognition with spatiotemporal 3d convolutions. In: 2018 24Th international conference on pattern recognition (ICPR). IEEE, pp 2516–2521

  25. Hara K, Kataoka H, Satoh Y (2018) Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6546–6555

  26. Harrou F, Zerrouki N, Sun Y, Houacine A (2017) Vision-based fall detection system for improving safety of elderly people. IEEE Instrument Measure Magaz 20(6):49–55

    Article  Google Scholar 

  27. He J, Bai S, Wang X (2017) An unobtrusive fall detection and alerting system based on kalman filter and bayes network classifier. Sensors 17(6):1393

    Article  Google Scholar 

  28. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  29. Hsieh CY, Liu KC, Huang CN, Chu WC, Chan CT (2017) Novel hierarchical fall detection algorithm using a multiphase fall model. Sensors 17(2):307

    Article  Google Scholar 

  30. Igual R, Medrano C, Plaza I (2013) Challenges, issues and trends in fall detection systems. Biomed Engi Online 12(1):66

    Article  Google Scholar 

  31. Jahanjoo A, Naderan M, Rashti MJ (2020) Detection and multi-class classification of falling in elderly people by deep belief network algorithms. Journal of Ambient Intelligence and Humanized Computing, 1–21

  32. Kangas M, Vikman I, Nyberg L, Korpelainen R, Lindblom J, Jämsä T (2012) Comparison of real-life accidental falls in older people with experimental falls in middle-aged test subjects. Gait Posture 35(3):500–505

    Article  Google Scholar 

  33. Khan SS, Taati B (2017) Detecting unseen falls from wearable devices using channel-wise ensemble of autoencoders. Expert Syst Appl 87:280–290

    Article  Google Scholar 

  34. Khan MS, Yu M, Feng P, Wang L, Chambers J (2015) An unsupervised acoustic fall detection system using source separation for sound interference suppression. Signal Processing 110:199–210

    Article  Google Scholar 

  35. Klenk J, Becker C, Lieken F, Nicolai S, Maetzler W, Alt W, Zijlstra W, Hausdorff J, Van Lummel R, Chiari L et al (2011) Comparison of acceleration signals of simulated and real-world backward falls. Med Eng Phys 33(3):368–373

    Article  Google Scholar 

  36. Le XH, Ho HV, Lee G, Jung S (2019) Application of long short-term memory (lstm) neural network for flood forecasting. Water 11(7):1387

  37. Li Z, Gavrilyuk K, Gavves E, Jain M, Snoek CG (2018) Videolstm convolves, attends and flows for action recognition. Comput Vis Image Underst 166:41–50

    Article  Google Scholar 

  38. Liu J, Tan R, Han G, Sun N, Kwong S (2020) Privacy-preserving in-home fall detection using visual shielding sensing and private information-embedding. IEEE Transactions on Multimedia

  39. Liu J, Xia Y, Tang Z (2020) Privacy-preserving video fall detection using visual shielding information. Vis Comput, 1–12

  40. Lowry CA, Woodall WH, Champ CW, Rigdon SE (1992) A multivariate exponentially weighted moving average control chart. Technometrics 34 (1):46–53

    Article  MATH  Google Scholar 

  41. Lu N, Wu Y, Feng L, Song J (2018) Deep learning for fall detection: Three-dimensional cnn combined with lstm on video kinematic data. IEEE J Biomed Health Inform 23(1):314–323

    Article  Google Scholar 

  42. Ma C, Shimada A, Uchiyama H, Nagahara H, Ri Taniguchi (2019) Fall detection using optical level anonymous image sensing system. Opt Laser Technol 110:44–61

    Article  Google Scholar 

  43. Manekar R, Saurav S, Maiti S, Singh S, Chaudhury S, Kumar R, Chaudhary K et al (2020) Activity recognition for indoor fall detection in 360-degree videos using deep learning techniques. In: Proceedings of 3rd international conference on computer vision and image processing. Springer, pp 417–429

  44. Mao A, Ma X, He Y, Luo J (2017) Highly portable, sensor-based system for human fall monitoring. Sensors 17(9):2096

    Article  Google Scholar 

  45. MarketsAndMarkets (2017) Fall detection system market by component (accelerometer & gyroscope, unimodal/bimodal, multimodal sensors), algorithm (simple threshold, machine learning), system (in- home landline, in-home cellular, wearable), end user, and region - global forecast to 2022. Tech. rep., MarketsAndMarkets. https://www.marketsandmarkets.com/Market-Reports/fall-detection-system-market-125162303.htmlhttps://www.marketsandmarkets.com/Market-Reports/fall-detection-system-market-125162303.html, Accessed 20 Nov 2020

  46. Martínez-Villaseñor L, Ponce H, Brieva J, Moya-Albor E, Núñez-Martínez J, Peñafort-Asturiano C (2019) Up-fall detection dataset: a multimodal approach. Sensors 19(9):1988

    Article  Google Scholar 

  47. Mauldin TR, Canby ME, Metsis V, Ngu AH, Rivera CC (2018) Smartfall: a smartwatch-based fall detection system using deep learning. Sensors 18 (10):3363

    Article  Google Scholar 

  48. Mozaffari N, Rezazadeh J, Farahbakhsh R, Yazdani S, Sandrasegaran K (2019) Practical fall detection based on iot technologies: a survey, vol 8

  49. Mubashir M, Shao L, Seed L (2013) A survey on fall detection: Principles and approaches. Neurocomputing 100:144–152

    Article  Google Scholar 

  50. Nait Aicha A, Englebienne G, Van Schooten KS, Pijnappels M, Kröse B (2018) Deep learning to predict falls in older adults based on daily-life trunk accelerometry. Sensors 18(5):1654

    Article  Google Scholar 

  51. Nations U et al (2017) World population ageing 2017: Highlights New York. Department of Economic and Social Affairs, United Nations

  52. Núñez-Marcos A, Azkune G, Arganda-Carreras I (2017) Vision-based fall detection with convolutional neural networks. Wireless Communications and Mobile Computing, 2017

  53. Organization WH et al (2017) Falls fact sheet. Diakses dari: http://wwwwhoint/mediacentre/factsheets/fs344/en/on November 29:2017

  54. Panahi L, Ghods V (2018) Human fall detection using machine vision techniques on rgb–d images. Biomed Signal Process Control 44:146–153

    Article  Google Scholar 

  55. Pierleoni P, Belli A, Palma L, Pellegrini M, Pernini L, Valenti S (2015) A high reliability wearable device for elderly fall detection. IEEE Sensors J 15(8):4544–4553

    Article  Google Scholar 

  56. Redd JL, Zura RD, Tanner AE, Walk EE, Wu MM (1992) Personal emergency response systems. J Burn Care Rehabilit 13(4):453–459

    Article  Google Scholar 

  57. Ricciuti M, Spinsante S, Gambi E (2018) Accurate fall detection in a top view privacy preserving configuration. Sensors 18(6):1754

    Article  Google Scholar 

  58. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vision 115(3):211–252

    Article  MathSciNet  Google Scholar 

  59. Sannino G, De Falco I, De Pietro G (2015) A supervised approach to automatically extract a set of rules to support fall detection in an mhealth system. Appl Soft Comput 34:205–216

    Article  Google Scholar 

  60. Santos GL, Endo PT, Monteiro KHDC, Rocha EDS, Silva I, Lynn T (2019) Accelerometer-based human fall detection using convolutional neural networks. Sensors 19(7):1644

    Article  Google Scholar 

  61. Saurav S, Kiran TM, Reddy BSK, Srivastav KS, Singh S, Saini R (2018) Dynamic image networks for human fall detection in 360-degree videos. In: Workshop on computer vision applications. Springer, pp 65–78

  62. Saurav S, Saini R, Singh S (2021) A dual-stream fused neural network for fall detection in multi-camera and 360∘ videos. Neural Comput & Applic, 1–28

  63. Shahzad A, Kim K (2018) Falldroid: an automated smart-phone-based fall detection system using multiple kernel learning. IEEE Trans Indust Inform 15(1):35–44

    Article  Google Scholar 

  64. Shrivastava R, Pandey M (2020) Real time fall detection in fog computing scenario. Clust Comput, 1–10

  65. Soomro K, Zamir AR, Shah M (2012) Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv:12120402

  66. Tahir A, Ahmad J, Morison G, Larijani H, Gibson RM, Skelton DA (2019) Hrnn4f: Hybrid deep random neural network for multi-channel fall activity detection. Probability in the Engineering and Informational Sciences, 1–14

  67. Torti E, Fontanella A, Musci M, Blago N, Pau D, Leporati F, Piastra M (2019) Embedding recurrent neural networks in wearable systems for real-time fall detection. Microprocess Microsyst 71:102895

    Article  Google Scholar 

  68. Tran D, Bourdev L, Fergus R, Torresani L, Paluri M (2015) Learning spatiotemporal features with 3d convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp 4489–4497

  69. Tran TH, Le TL, Hoang VN, Vu H (2017) Continuous detection of human fall using multimodal features from kinect sensors in scalable environment. Comput Methods Programs Biomed 146:151–165

  70. Ullah A, Ahmad J, Muhammad K, Sajjad M, Baik SW (2017) Action recognition in video sequences using deep bi-directional lstm with cnn features. IEEE Access 6:1155–1166

    Article  Google Scholar 

  71. Wang S, Chen L, Zhou Z, Sun X, Dong J (2016) Human fall detection in surveillance video based on pcanet. Multimed Tools Appl 75 (19):11603–11613

    Article  Google Scholar 

  72. Wang L, Xu Y, Cheng J, Xia H, Yin J, Wu J (2018) Human action recognition by learning spatio-temporal features with deep neural networks. IEEE Access 6:17913–17922

    Article  Google Scholar 

  73. Wu F, Zhao H, Zhao Y, Zhong H (2015) Development of a wearable-sensor-based fall detection system. Int J Telemed Appl

  74. Xie J, Chen B, Gu X, Liang F, Xu X (2019) Self-attention-based bilstm model for short text fine-grained sentiment classification. IEEE Access 7:180558–180570

    Article  Google Scholar 

  75. Xingjian S, Chen Z, Wang H, Yeung DY, Wong WK, Woo Wc (2015) Convolutional lstm network: A machine learning approach for precipitation nowcasting. In: Advances in neural information processing systems, pp 802–810

  76. Yao C, Hu J, Min W, Deng Z, Zou S, Min W (2020) A novel real-time fall detection method based on head segmentation and convolutional neural network. J Real-Time Image Proc 17:1939–1949

    Article  Google Scholar 

  77. Yao L, Yang W, Huang W (2020) A fall detection method based on a joint motion map using double convolutional neural networks. Multimed Tools Appl, 1–18

  78. Zerrouki N, Houacine A (2018) Combined curvelets and hidden markov models for human fall detection. Multimed Tools Appl 77(5):6405–6424

    Article  Google Scholar 

  79. Zhang Z, Ma X, Wu H, Li Y (2018) Fall detection in videos with trajectory-weighted deep-convolutional rank-pooling descriptor. IEEE Access 7:4135–4144

    Article  Google Scholar 

  80. Zhang Q, Ren L, Shi W (2013) Honey: a multimodality fall detection and telecare system. Telemed E-Health 19(5):415–429

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the director, CSIR-CEERI, Pilani for supporting and encouraging research activities at CSIR-CEERI, Pilani. Constant motivation by the group head, cognitive computing group, CSIR-CEERI is also acknowledged.

Author information

Authors and Affiliations

  1. Academy of Scientific & Innovative Research, Ghaziabad, India

    Sumeet Saurav, Ravi Saini & Sanjay Singh

  2. CSIR-Central Electronics Engineering Research Institute, Pilani, India

    Sumeet Saurav, Ravi Saini & Sanjay Singh

Authors
  1. Sumeet Saurav
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ravi Saini
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sanjay Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sumeet Saurav.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saurav, S., Saini, R. & Singh, S. Vision-based techniques for fall detection in 360∘ videos using deep learning: Dataset and baseline results. Multimed Tools Appl 81, 14173–14216 (2022). https://doi.org/10.1007/s11042-022-12366-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12366-5

Keywords

Search

Navigation