Skip to main content
SpringerLink
Log in

Elderly fall detection based on multi-stream deep convolutional networks

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Fall is the biggest threat to seniors, with significant emotional, physical and financial implications. It is the major cause of serious injuries, disabilities, hospitalizations and even death especially for elderly people living alone. Timely detection could provide immediate medical service to the injured and avoid its harmful consequences. Great number of vision-based techniques has been proposed by installing cameras in several everyday environments. Recently, deep learning has revolutionized these techniques, mostly using convolutional neural networks (CNNs). In this paper, we propose weighted multi-stream deep convolutional neural networks that exploit the rich multimodal data provided by RGB-D cameras. Our method detects automatically fall events and sends a help request to the caregivers. Our contribution is three-fold. We build a new architecture composed of four separate CNN streams, one for each modality. The first modality is based on a single combined RGB and depth image to encode static appearance information. RGB image is used to capture color and texture and depth image deals with illumination variations. In contrast of the first feature that lacks the contextual information about previous and next frames, the second modality characterizes the human shape variations. After background subtraction and person recognition, human silhouette is extracted and stacked to define history of binary motion HBMI. The last two modalities are used to more discriminate the motion information. Stacked amplitude and oriented flow are used in addition to stacked optical flow field to describe respectively the velocity, the direction and the motion displacements. The main motivation behind the use of these multimodal data is to combine complementary information such as motion, shape, RGB and depth appearance to achieve more accurate detection than using only one modality. Our second contribution is the combination of the four streams to generate the final decision for fall detection. We evaluate early and late fusion strategies and we have defined the weight of each modality based on its overall system performance. Weighted score fusion is finally adopted based on our experiments. In the third contribution, transfer learning and data augmentation are applied to increase the amount of training data, avoid over fitting and improve the accuracy. Experiments have been conducted on publicly available standard datasets and demonstrate the effectiveness of the proposed method compared to existing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Adam MA, Azkune G, Arganda-Carreras I (2017) Vision-based fall detection with convolutional neural networks. Wirel Commun Mob Comput 1:1–16

    Google Scholar 

  2. Adhikari K, Bouchachia H, Nait-Charif H (2017) Activity recognition for indoor fall detection using convolutional neural network. In Proceedings of the 15th IAPR International Conference on Machine Vision Applications, Nagoya, Japan, pp. 81–84

  3. Albawendi S, Lotfi A, Powell H, & Appiah K (2018). Video based fall detection using features of motion, shape and histogram. Proceedings of the 11th PErvasive technologies related to assistive environments conference on - PETRA ‘18. doi:10.1145/3197768.3201539

  4. Alhimale L, Zedan H, Al-Bayatti A (2014) The implementation of an intelligent and video-based fall detection system using a neural network. Appl Soft Comput 18:59–69. https://doi.org/10.1016/j.asoc.2014.01.024

    Article  Google Scholar 

  5. Allali G, Ayers EI, Holtzer R, Verghese J (2017) The role of postural instability/gait difficulty and fear of falling in predicting falls in non-demented older adults. Arch Gerontol Geriatr 69:15–20

    Article  Google Scholar 

  6. Barnich O, Van Droogenbroeck M (2011) ViBe: a universal background subtraction algorithm for video sequences. IEEE Trans Image Process 20(6):1709–1724

    Article  MathSciNet  Google Scholar 

  7. Burns ER, Stevens JA, Lee R (2016) (2016). The direct costs of fatal and non-fatal falls among older adults — United States. J Saf Res 58:99–103

    Article  Google Scholar 

  8. Casilari E, Oviedo-Jiménez MA. (2015) Automatic fall detection system based on the combined use of a smartphone and a smartwatch. PLoS one 2015; 10(11): e0140929.

  9. Charfi I, Miteran J, Dubois J, Atri M, Tourki R (2013) Optimized spatio-temporal descriptors for real-time fall detection: comparison of support vector machine and Adaboost-based classification. Journal of Electronic Imaging 22(4):041106. https://doi.org/10.1117/1.jei.22.4.041106

    Article  Google Scholar 

  10. Chua JL, Chang YC, Lim WK (2013) “A simple vision-based fall detection technique for indoor video surveillance”. Signal, Image and Video Processing: p. 1–11

  11. Cippitelli E, Fioranelli F, Gambi E, Spinsante S (2017) Radar and RGB-depth sensors for fall detection: a review. IEEE Sensors J 17(12):3585–3604

    Article  Google Scholar 

  12. Colque RVHM, Junior CAC, Schwartz WR (2015) Histograms of optical flow orientation and magnitude to detect anomalous events in videos. 2015 28th SIBGRAPI

  13. Delahoz Y, Labrador M (2014) Survey on fall detection and fall prevention using wearable and external sensors. Sensors 14:19806–19842

    Article  Google Scholar 

  14. Dollar P, Belongie S, Perona P (2010) The fastest pedestrian detector in the west, in: British Machine Vision Conference, pp. 68.1–68.11

  15. Doulamis A, Doulamis N (2018) Adaptive deep learning for a vision-based fall detection. Proceedings of the 11th PETRA ‘18

  16. Espinosa R, Ponce H, Gutiérrez S, Martínez-Villaseñor L, Brieva, J., & Moya-Albor, E. (2019). A vision-based approach for fall detection using multiple cameras and convolutional neural networks: A case study using the UP-Fall detection dataset Computers in Biology and Medicine, 103520. doi:https://doi.org/10.1016/j.compbiomed.2019.103520

  17. Fan Y, Levine MD, Wen G, Qiu S (2017) A deep neural network for real-time detection of falling humans in naturally occurring scenes. Neurocomputing 260:43–58

    Article  Google Scholar 

  18. Gasparrini S, Cippitelli E, Spinsante S, Gambi E (2014) A depth-based fall detection system using a Kinect sensor. Sensors 14:2756–2775

    Article  Google Scholar 

  19. Gunner Farneback (2003) “Two-Frame Motion Estimation Based on Polynomial Expansion”, Image Analysis, pp363–370

  20. Kepski, M.; Kwolek, B (2014) Fall detection using ceiling-mounted 3D depth camera. In Proceedings of the 2014 International conference on computer vision theory and applications (VISAPP), Lisbon, Portugal, 5–8 January 2014; Volume 2, pp. 640–647

  21. Kharrat O, Mersni E, Guebsi O, Ben Salah FZ, Dziri C (2017) Qualité de vie et personnes âgées en Tunisie, NPG Neurologie - Psychiatrie - Gériatrie, Volume 17, Issue 97, Pages 5–11

  22. Khraief C, Benzarti F, Amiri H (2017) Multi person detection and tracking based on hierarchical Level-Set method. In: The 10th International Conference on Machine Vision (ICMV) pp.91–98

  23. Khraief C, Benzarti F, Amiri H (2018) Vision-based fall detection for elderly people using body parts movement and shape analysis. In: The 10th International Conference on Machine Vision (ICMV)

  24. Kistler BM, Khubchandani J, Jakubowicz G, Wilund K, Sosnoff J. (2018). Falls and Fall-Related Injuries Among US Adults Aged 65 or Older With Chronic Kidney Disease. Prev Chronic Dis 2018;15:170518.

  25. Kwolek B, Kepski M (2014) Human fall detection on embedded platform using depth maps and wireless accelerometer. Comput Methods Prog Biomed 117:489–501

    Article  Google Scholar 

  26. Lie WN, Le AT, Lin GH (2018) Human fall-down event detection based on 2D skeletons and deep learning approach. In Proceedings of the International Workshop on Advanced Image Technology, Chiang Mai, Thailand, pp. 1–4

  27. Lin HY, Hsueh YL, Lie WN (2016) Abnormal event detection using Microsoft kinect in a smart home. In Proceedings of the 2016 International Computer Symposium, Chiayi, Taiwan, pp. 285–289

  28. Liu J, Zha ZJ, Tian Q, Liu D, Yao T, Ling Q, Mei T (2016) Multi-scale triplet cnn for person re-identification. In: Proceedings of the 2016 ACM on Multimedia Conference, MM ‘16, pp. 192–196. ACM, New York, NY, USA. https://doi.org/10.1145/2964284.2967209

  29. Lu N, Wu Y, Feng L, Song J (2019) Deep learning for fall detection: three-dimensional CNN combined with LSTM on video kinematic data. IEEE journal of biomedical and health Informatics, 23(1), 314-323. doi:https://doi.org/10.1109/jbhi.2018.2808281

  30. Mastorakis G, Makris D (2012) Fall detection system using Kinect’s infrared sensor. J Real-Time Image Process 9:635–646. https://doi.org/10.1007/s11554-012-0246

    Article  Google Scholar 

  31. Mauldin TR, Canby ME, Metsis V, Ngu AHH, Rivera CC (2018) SmartFall: A Smartwatch-Based Fall Detection System Using Deep Learning. Sensors (Basel). 18(10):3363. Published 2018 Oct 9. doi:10.3390/s18103363

  32. Merrouche F, Baha N (2017) ” Fall Detection using Head Tracking and Centroid Movement Based on a Depth Camera”, International Conference ICCES pp. 29–3

  33. Mubashir M, Shao L, Seed L (2013) A survey on fall detection: principles and approaches. Neurocomputing 100:144–152

    Article  Google Scholar 

  34. Nizam Y, Mohd MNH, Jamil MMA (2017) Human fall detection from depth images using position and velocity of subject. Procedia Computer Science 105, 131-137

  35. Nizam Y, Mohd M, Jamil M (2018) Development of a user-adaptable human fall detection based on fall risk levels using depth sensor. Sensors 18(7):2260

    Article  Google Scholar 

  36. Noury N, et al (2007) “Fall detection principles and methods.” 2007 29th Annual International Conferenceof the IEEE Engineering in Medicine and Biology Society. IEEE

  37. Pathak D, Bhosale VK (2017) Fall detection for elderly people in homes using Kinect sensor. Int. J. Innov. Res. Comput. Commun. Eng. 5, 1468–1474. Doi:https://doi.org/10.15680/IJIRCCE.2017

  38. Rahman MM, Tan Y, Xue J, Lu K (2017) RGB-D object recognition with multimodal deep convolutional neural networks. In Proceedings of the IEEE International Conference on Multimedia and Expo., Hong Kong, China, 10–14, pp. 991–996

  39. Rougier C, Meunier J, St-Arnaud A, and Rousseau J (2007) “Fall detection from human shape and motion history using video surveillance,” in Proceedings of AINAW’07, pp. 875–880

  40. Roy N, Dubé R, Després C, Freitas A, Légaré F (2018) Choosing between staying at home or moving: a systematic review of factors influencing housing decisions among frail older adults. PLoS One 13(1):e0189266. https://doi.org/10.1371/journal.pone.0189266

    Article  Google Scholar 

  41. Santos GL, Endo PT, Monteiro KHC, Rocha EDS, Silva I, Lynn T (2019) Accelerometer-Based Human Fall Detection Using Convolutional Neural Networks. Sensors (Basel). 19(7):1644. Published 2019 Apr 6. doi:10.3390/s19071644.

  42. Sehairi K, Chouireb F, Meunier J (2018) Elderly fall detection system based on multiple shape features and motion analysis. 2018 international conference on intelligent systems and computer vision

  43. Simonyan and Zisserman A (2014) Very deep convolutional networks for large-scale image recognition, arXiv 1409.1556 , 2014

  44. Soomro K, Zamir AR and Shah M (2012) UCF101: A Dataset of 101 Human Action Classes From Videos in The Wild.CRCV-TR-12-01

  45. Stevens JA, Rudd RA (2015) Circumstances and contributing causes of fall deaths among persons aged 65 and older, J. Am. Geriatr. Soc., Vol. 62, No. 3, pp. 470–475, 2015.

  46. Uddin M, Khaksar W, Torresen J (2018) Ambient sensors for elderly care and independent living: a survey. Sensors 18(7):2027. https://doi.org/10.3390/s18072027

    Article  Google Scholar 

  47. Vielzeuf V, Pateux S, and Jurie F (2017) Temporal mul-timodal fusion for video emotion classification in the wild. InProceedingsof the 19th ACM International Conference on Multimodal Interaction,pages 569–576. ACM

  48. Wang L, Xiong Y, Wang Z, Qiao Y, Lin D, Tang X, and Van Gool L (2016) Temporal segment networks: Towards goodpractices for deep action recognition. InEuropean Confer-ence on Computer Vision, pages 20–36. Springer

  49. Wang S, Chen L, Zhou Z, Sun X, Dong J (2016) Human fall detection in surveillance video based on PCANet. Multimed Tools Appl 75(19):11603–11613

    Article  Google Scholar 

  50. Wang K, Cao G, Meng D, Chen W, and Cao W (2016) “Automatic fall detection of human in video using combination of features,” IEEE International Conference BIBM, pp. 1228–1233,China

  51. Wang W, Lu X, Song J, Chen C (2016) A two-column convolutional neural network for facial point detection. In: 2016 International Conference on Progress in Informatics and Computing (PIC), pp. 169–173. https://doi.org/10.1109/PIC.2016.7949488

  52. Wortmann M (2012) Dementia: a global health priority - highlights from an ADI and World Health. Organization report. Alzheimers Res Ther 4(5):40

    Article  Google Scholar 

  53. Xu T, Zhou Y, Zhu J (2018) New advances and challenges of fall detection systems: a survey. Appl Sci 8:418. https://doi.org/10.3390/app8030418

    Article  Google Scholar 

  54. Yang X, Molchanov P, and Kautz J (2016) Multilayer and mul-timodal fusion of deep neural networks for video classification. InPro-ceedings of the 2016 ACM on multimedia conference, pages 978–987.ACM

  55. Yu M, Rhuma A, Naqvi SM, Wang L, Chambers J (2012) A posture recognition-based fall detection system for monitoring an elderly person in a smart home environment. IEEE Trans Inf Technol Biomed 16(6):1274–1286

    Article  Google Scholar 

  56. Yun Y, Gu IY-H (2016) Human fall detection in videos via boosting and fusing statistical features of appearance, shape and motion dynamics on Riemannian manifolds with applications to assisted living. Comput Vis Image Underst 148:111–122

    Article  Google Scholar 

  57. Zerrouki N, Houacine A (2017) Combined curvelets and hidden Markov models for human fall detection. Multimed Tools Appl 77(5):6405–6424

    Article  Google Scholar 

  58. Zhang Y, Zhou D, Chen S, Gao S, Ma Y (2016) Single-image crowd counting via multi-column convolutional neural network. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 589–597. https://doi.org/10.1109/CVPR.2016.70

  59. Zhong Z, Christopher C, Vassilis A (2015) A survey on vision-based fall detection, Proceedings of the 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments, Corfu, Greece, doi: 10.1145/2769493.2769540

  60. Zhou X, Qian L-C, You P-J, Ding Z-G, Han Y-Q (2018) Fall detection using convolutional neural network with multi-sensor fusion. 2018 IEEE international conference on Multimedia & Expo Workshops (ICMEW). https://doi.org/10.1109/icmew.2018.8551564

Download references

Author information

Authors and Affiliations

  1. SITI Laboratory, National Engineering School of Tunis (ENIT), University of Tunis El Manar, Tunis, Tunisia

    Chadia Khraief, Faouzi Benzarti & Hamid Amiri

Authors
  1. Chadia Khraief
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Faouzi Benzarti
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hamid Amiri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chadia Khraief.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khraief, C., Benzarti, F. & Amiri, H. Elderly fall detection based on multi-stream deep convolutional networks. Multimed Tools Appl 79, 19537–19560 (2020). https://doi.org/10.1007/s11042-020-08812-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-08812-x

Keywords

Search

Navigation