Skip to main content
SpringerLink
Log in

Fusion of Multimodal Sensor Data for Effective Human Action Recognition in the Service of Medical Platforms

  • Conference paper
  • First Online:
MultiMedia Modeling (MMM 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12573))

Included in the following conference series:

Abstract

In what has arguably been one of the most troubling periods of recent medical history, with a global pandemic emphasising the importance of staying healthy, innovative tools that shelter patient well-being gain momentum. In that view, a framework is proposed that leverages multimodal data, namely inertial and depth sensor-originating data, can be integrated in health care-oriented platforms, and tackles the crucial task of human action recognition (HAR). To analyse person movement and consequently assess the patient’s condition, an effective methodology is presented that is two-fold: initially, Kinect-based action representations are constructed from handcrafted 3DHOG depth features and the descriptive power of a Fisher encoding scheme. This is complemented by wearable sensor data analysis, using time domain features and then boosted by exploring fusion strategies of minimum expense. Finally, an extended experimental process reveals competitive results in a well-known benchmark dataset and indicates the applicability of our methodology for HAR.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.gatekeeper-project.eu/.

References

  1. Avci, A., Bosch, S., Marin-Perianu, M., Marin-Perianu, R., Havinga, P.: Activity recognition using inertial sensing for healthcare, wellbeing and sports applications: a survey. In: 23th International Conference on Architecture of Computing Systems 2010, pp. 1–10. VDE (2010)

    Google Scholar 

  2. Benser, E.T.: Trends in inertial sensors and applications. In: 2015 IEEE International Symposium on Inertial Sensors and Systems (ISISS) Proceedings, pp. 1–4 (2015)

    Google Scholar 

  3. Chen, C., Jafari, R., Kehtarnavaz, N.: Improving human action recognition using fusion of depth camera and inertial sensors. IEEE Trans. Hum.-Mach. Syst. 45(1), 51–61 (2015)

    Article  Google Scholar 

  4. Chen, C., Jafari, R., Kehtarnavaz, N.: A real-time human action recognition system using depth and inertial sensor fusion. IEEE Sens. J. 16(3), 773–781 (2015)

    Article  Google Scholar 

  5. Chen, C., Jafari, R., Kehtarnavaz, N.: UTD-MHAD: a multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor. In: 2015 IEEE International Conference on Image Processing (ICIP), pp. 168–172. IEEE (2015)

    Google Scholar 

  6. Chen, C., Jafari, R., Kehtarnavaz, N.: A survey of depth and inertial sensor fusion for human action recognition. Multimedia Tools Appl. 76(3), 4405–4425 (2015). https://doi.org/10.1007/s11042-015-3177-1

    Article  Google Scholar 

  7. Chen, C., Liu, M., Zhang, B., Han, J., Jiang, J., Liu, H.: 3D action recognition using multi-temporal depth motion maps and fisher vector. In: IJCAI, pp. 3331–3337 (2016)

    Google Scholar 

  8. Chen, L., Wei, H., Ferryman, J.: A survey of human motion analysis using depth imagery. Pattern Recogn. Lett. 34(15), 1995–2006 (2013)

    Article  Google Scholar 

  9. Chen, Y., Le, D., Yumak, Z., Pu, P.: EHR: a sensing technology readiness model for lifestyle changes. Mob. Netw. Appl. 22(3), 478–492 (2017)

    Article  Google Scholar 

  10. Collin, J., Davidson, P., Kirkko-Jaakkola, M., Leppäkoski, H.: Inertial sensors and their applications. In: Bhattacharyya, S.S., Deprettere, E.F., Leupers, R., Takala, J. (eds.) Handbook of Signal Processing Systems, pp. 51–85. Springer, Cham (2019). https://doi.org/10.1007/978-3-319-91734-4_2

    Chapter  Google Scholar 

  11. Dawar, N., Ostadabbas, S., Kehtarnavaz, N.: Data augmentation in deep learning-based fusion of depth and inertial sensing for action recognition. IEEE Sens. Lett. 3(1), 1–4 (2019)

    Article  Google Scholar 

  12. Dawar, N., Ostadabbas, S., Kehtarnavaz, N.: Data augmentation in deep learning-based fusion of depth and inertial sensing for action recognition. IEEE Sens. Lett. 3(1), 1–4 (2018)

    Article  Google Scholar 

  13. Delachaux, B., Rebetez, J., Perez-Uribe, A., Satizábal Mejia, H.F.: Indoor activity recognition by combining one-vs.-all neural network classifiers exploiting wearable and depth sensors. In: Rojas, I., Joya, G., Cabestany, J. (eds.) IWANN 2013. LNCS, vol. 7903, pp. 216–223. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38682-4_25

    Chapter  Google Scholar 

  14. Ehatisham-Ul-Haq, M., et al.: Robust human activity recognition using multimodal feature-level fusion. IEEE Access 7, 60736–60751 (2019)

    Article  Google Scholar 

  15. Elmadany, N.E.D., He, Y., Guan, L.: Human action recognition using hybrid centroid canonical correlation analysis. In: 2015 IEEE International Symposium on Multimedia (ISM), pp. 205–210. IEEE (2015)

    Google Scholar 

  16. Kwolek, B., Kepski, M.: Human fall detection on embedded platform using depth maps and wireless accelerometer. Comput. Methods Programs Biomed. 117(3), 489–501 (2014)

    Article  Google Scholar 

  17. Lane, N.D., et al.: Bewell: sensing sleep, physical activities and social interactions to promote wellbeing. Mob. Netw. Appl. 19(3), 345–359 (2014)

    Article  MathSciNet  Google Scholar 

  18. Lara, O.D., Labrador, M.A.: A survey on human activity recognition using wearable sensors. IEEE Commun. Surv. Tutor. 15(3), 1192–1209 (2012)

    Article  Google Scholar 

  19. Li, W., Zhang, Z., Liu, Z.: Action recognition based on a bag of 3D points. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 9–14. IEEE (2010)

    Google Scholar 

  20. Liu, K., Chen, C., Jafari, R., Kehtarnavaz, N.: Fusion of inertial and depth sensor data for robust hand gesture recognition. IEEE Sens. J. 14(6), 1898–1903 (2014)

    Article  Google Scholar 

  21. Liu, L., Shao, L.: Learning discriminative representations from RGB-D video data. In: Twenty-Third International Joint Conference on Artificial Intelligence (2013)

    Google Scholar 

  22. Masum, A.K.M., Bahadur, E.H., Shan-A-Alahi, A., Uz Zaman Chowdhury, M.A., Uddin, M.R., Al Noman, A.: Human activity recognition using accelerometer, gyroscope and magnetometer sensors: deep neural network approaches. In: 2019 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT), pp. 1–6 (2019)

    Google Scholar 

  23. Mavropoulos, T., et al.: A smart dialogue-competent monitoring framework supporting people in rehabilitation. In: Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments, pp. 499–508 (2019)

    Google Scholar 

  24. Munson, S.A., Consolvo, S.: Exploring goal-setting, rewards, self-monitoring, and sharing to motivate physical activity. In: 2012 6th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) and Workshops, pp. 25–32. IEEE (2012)

    Google Scholar 

  25. Ramasamy Ramamurthy, S., Roy, N.: Recent trends in machine learning for human activity recognition-a survey. Wiley Interdisc. Rev. Data Mining Knowl. Discov. 8(4), e1254 (2018)

    Article  Google Scholar 

  26. Shaeffer, D.K.: Mems inertial sensors: a tutorial overview. IEEE Commun. Mag. 51(4), 100–109 (2013)

    Article  Google Scholar 

  27. Sidor, K., Wysocki, M.: Recognition of human activities using depth maps and the viewpoint feature histogram descriptor. Sensors 20(10), 2940 (2020)

    Article  Google Scholar 

  28. Uijlings, J.R., Duta, I.C., Rostamzadeh, N., Sebe, N.: Realtime video classification using dense HOF/HOG. In: Proceedings of International Conference on Multimedia Retrieval, pp. 145–152 (2014)

    Google Scholar 

  29. Wang, H., Schmid, C.: Action recognition with improved trajectories. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2013

    Google Scholar 

  30. Wang, J., Liu, Z., Wu, Y., Yuan, J.: Mining actionlet ensemble for action recognition with depth cameras. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1290–1297. IEEE (2012)

    Google Scholar 

  31. Wang, J., Chen, Y., Hao, S., Peng, X., Hu, L.: Deep learning for sensor-based activity recognition: a survey. Pattern Recogn. Lett. 119, 3–11 (2019)

    Article  Google Scholar 

  32. Weiyao, X., Muqing, W., Min, Z., Yifeng, L., Bo, L., Ting, X.: Human action recognition using multilevel depth motion maps. IEEE Access 7, 41811–41822 (2019)

    Article  Google Scholar 

  33. Wong, C., McKeague, S., Correa, J., Liu, J., Yang, G.Z.: Enhanced classification of abnormal gait using BSN and depth. In: 2012 Ninth International Conference on Wearable and Implantable Body Sensor Networks, pp. 166–171. IEEE (2012)

    Google Scholar 

  34. Zhang, B., Yang, Y., Chen, C., Yang, L., Han, J., Shao, L.: Action recognition using 3D histograms of texture and a multi-class boosting classifier. IEEE Trans. Image Process. 26(10), 4648–4660 (2017)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgment

This research has been financed by the European Regional Development Fund of the European Union and Greek national funds through the Operational Program Competitiveness, Entrepreneurship and Innovation, under the call RESEARCH - CREATE - INNOVATE (T1EDK-00686) and the EC funded project GATEKEEPER (H2020-857223).

Author information

Authors and Affiliations

  1. Centre for Research and Technology Hellas, Information Technologies Institute, Thessaloniki, Greece

    Panagiotis Giannakeris, Athina Tsanousa, Thanasis Mavropoulos, Georgios Meditskos, Konstantinos Ioannidis, Stefanos Vrochidis & Ioannis Kompatsiaris

Authors
  1. Panagiotis Giannakeris
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Athina Tsanousa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thanasis Mavropoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Georgios Meditskos
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Konstantinos Ioannidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Stefanos Vrochidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Ioannis Kompatsiaris
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Panagiotis Giannakeris .

Editor information

Editors and Affiliations

  1. Charles University, Prague, Czech Republic

    Jakub Lokoč

  2. Charles University, Prague, Czech Republic

    Tomáš Skopal

  3. Klagenfurt University, Klagenfurt, Austria

    Klaus Schoeffmann

  4. CERTH-ITI, Thessaloniki, Greece

    Vasileios Mezaris

  5. Renmin University of China, Beijing, China

    Xirong Li

  6. CERTH-ITI, Thessaloniki, Greece

    Stefanos Vrochidis

  7. Queen Mary University of London, London, UK

    Ioannis Patras

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Giannakeris, P. et al. (2021). Fusion of Multimodal Sensor Data for Effective Human Action Recognition in the Service of Medical Platforms. In: Lokoč, J., et al. MultiMedia Modeling. MMM 2021. Lecture Notes in Computer Science(), vol 12573. Springer, Cham. https://doi.org/10.1007/978-3-030-67835-7_31

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-67835-7_31

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-67834-0

  • Online ISBN: 978-3-030-67835-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation