Skip to main content

Advertisement

SpringerLink
Log in

A deep-CNN based low-cost, multi-modal sensing system for efficient walking activity identification

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The misclassification of human activity information by IoT-based smart, health monitoring devices raises concerns about their reliability. For instance, the identification of walking activity by wrist-worn, hand-held, or pocketed devices is often error-prone because tracking of actual leg movement is necessary for proper identification of a walk. In contrast, wearable systems on the waist recognize the bodily movement as a whole and not limb-specific movement. In this paper, we consider the novel problem of distinguishing the walking activity defined as a sequence of repeated leg-swing actions from repetitive leg-swing activities in sitting state. For this purpose, a heterogeneous sensor system is implemented that acquires novel multi-modal data from low-cost leg-worn IMU sensors (m-module) and finger-tip based pulse sensors (p-module). This dataset is then processed to extract features for a supervised learning framework. A 1-D Deep Convolutional Neural Network (DCNN) model is used for system performance evaluation, that prevents misclassification of walking activity with an average accuracy of 97% and maximum accuracy of 99%. With the fusion of features, the DCNN model performs with 2% more accuracy than the performance of the system using only IMU data. For performance comparison with the DCNN model, 4 other supervised learning algorithms have been used, namely Support Vector Machine (SVM), k-Nearest Neighbors (kNN), Decision Tree (DT), and Gaussian Naive Bayes (GNB). The proposed DCNN model outperforms these algorithms by using the m-module features individually and in unison with the p-module features. The performance is also comparable to the state-of-the-art works on intelligent walking activity detection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24

Similar content being viewed by others

Notes

  1. https://www.electronicscomp.com/arduino-nano-3.0-development-board

  2. https://www.electronicscomp.com/pulse-sensor-india

  3. https://robu.in/product/mpu6050gyrosensor2accelerometer

  4. https://scikit-learn.org/stable/

  5. https://keras.io/

References

  1. Ahmed N, Rafiq JI, Islam MR (2020) Enhanced human activity recognition based on smartphone sensor data using hybrid feature selection model. Sensors 20(1):317

    Article  Google Scholar 

  2. Albright RK, Goska BJ, Hagen TM, Chi MY, Cauwenberghs G, Chiang PY (2011) Olam: A wearable, non-contact sensor for continuous heart-rate and activity monitoring. In: 2011 Annual international conference of the ieee engineering in medicine and biology society, pp 5625–5628. IEEE

  3. Altman NS (1992) An introduction to kernel and nearest-neighbor nonparametric regression. Am Stat 46(3):175–185

    MathSciNet  Google Scholar 

  4. Altun K, Barshan B (2010) Human activity recognition using inertial/magnetic sensor units. In: International workshop on human behavior understanding, pp 38–51. Springer

  5. Anguita D, Ghio A, Oneto L, Parra X, Reyes-Ortiz JL (2012) Human activity recognition on smartphones using a multiclass hardware-friendly support vector machine. In: International workshop on ambient assisted living, pp 216–223. Springer

  6. Attal F, Mohammed S, Dedabrishvili M, Chamroukhi F, Oukhellou L, Amirat Y (2015) Physical human activity recognition using wearable sensors. Sensors 15(12):31314–31338

    Article  Google Scholar 

  7. Avci A, Bosch S, Marin-Perianu M, Marin-Perianu R, Havinga P (2010) Activity recognition using inertial sensing for healthcare, wellbeing and sports applications: A survey. In: 23th International conference on architecture of computing systems 2010, pp 1–10. VDE

  8. Barshan B, Yüksek MC (2014) Recognizing daily and sports activities in two open source machine learning environments using body-worn sensor units. The Comput J 57(11):1649–1667

    Article  Google Scholar 

  9. Bhatti UA, Huang M, Wang H, Zhang Y, Mehmood A, Di W (2018) Recommendation system for immunization coverage and monitoring. Human Vaccines & Immunotherapeutics 14(1):165–171

    Article  Google Scholar 

  10. Bhatti UA, Huang M, Wu D, Zhang Y, Mehmood A, Han H (2019) Recommendation system using feature extraction and pattern recognition in clinical care systems. Enterp Inf Syst 13(3):329–351

    Article  Google Scholar 

  11. Bracewell R (1965) The autocorrelation function. The Fourier Transform and Its Applications, 40–45

  12. Bracewell RN, Bracewell RN (1986) The fourier transform and its applications. vol. 31999 McGraw-Hill

  13. Breiman L, Friedman J, Olshen R, Stone C (1984) Classification and regression trees (wadsworth & brooks/cole advanced books & software. Monterey, CA

  14. Chakraborty A, Mukherjee N (2022) A low-cost imu-based wearable system for precise identification of walk activity using deep convolutional neural network. In: Big data intelligence for smart applications, pp 117–140. Springer

  15. Chandra I, Sivakumar N, Gokulnath CB, Parthasarathy P (2019) Iot based fall detection and ambient assisted system for the elderly. Clust Comput 22(1):2517–2525

    Article  Google Scholar 

  16. Chen M-D, Kuo C-C, Pellegrini CA, Hsu M-J (2016) Accuracy of wristband activity monitors during ambulation and activities. Med Sci Sports Exerc 48(10):1942–1949

    Article  Google Scholar 

  17. Chetty G, White M, Akther F (2015) Smart phone based data mining for human activity recognition. Procedia Comput Sci 46:1181–1187

    Article  Google Scholar 

  18. Cho Y, Nam Y, Choi Y-J, Cho W-D (2008) Smartbuckle: human activity recognition using a 3-axis accelerometer and a wearable camera. In: Proceedings of the 2nd international workshop on systems and networking support for health care and assisted living environments, pp 1–3

  19. Chung W-Y, Lee Y-D, Jung S-J (2008) A wireless sensor network compatible wearable u-healthcare monitoring system using integrated ecg, accelerometer and spo 2. In: 2008 30th Annual international conference of the IEEE engineering in medicine and biology society, pp 1529–1532. IEEE

  20. Cleland I, Kikhia B, Nugent C, Boytsov A, Hallberg J, Synnes K, McClean S, Finlay D (2013) Optimal placement of accelerometers for the detection of everyday activities. Sensors 13(7):9183–9200

    Article  Google Scholar 

  21. Cooley JW, Tukey JW (1965) An algorithm for the machine calculation of complex fourier series. Math Comput 19(90):297–301

    Article  MathSciNet  MATH  Google Scholar 

  22. Department SR (2020) Smart Home Statistics & Facts. (Accessed 15 July 2020). https://www.statista.com/topics/2430/smart-homes

  23. Dernbach S, Das B, Krishnan NC, Thomas BL, Cook DJ (2012) Simple and complex activity recognition through smart phones. In: 2012 Eighth international conference on intelligent environments, pp 214–221. IEEE

  24. Fan L, Wang Z, Wang H (2013) Human activity recognition model based on decision tree. In: 2013 International conference on advanced cloud and big data, pp 64–68. IEEE

  25. Forecast wearables unit shipments worldwide from 2014 to 2023. https://www.statista.com/statistics/437871/wearables-worldwide-shipments/ (2020 accessed Accessed 25 Jan 2020)

  26. Gunn SR, et al. (1998) Support vector machines for classification and regression. ISIS Tech Rep 14(1):5–16

    Google Scholar 

  27. Hand DJ, Yu K (2001) Idiot’s bayes—not so stupid after all?. Int Stat Rev 69(3):385–398

    MATH  Google Scholar 

  28. Hardegger M, Roggen D, Tröster G (2015) 3d actionslam: wearable person tracking in multi-floor environments. Pers Ubiquit Comput 19(1):123–141

    Article  Google Scholar 

  29. Hegde N, Sazonov ES (2015) Smartstep 2.0-a completely wireless, versatile insole monitoring system. In: 2015 IEEE international conference on bioinformatics and biomedicine (BIBM), pp 746–749. IEEE

  30. Hinton GE, Krizhevsky A, Sutskever I, Srivastva N (2016) System and method for addressing overfitting in a neural network. Google Patents. US Patent 9,406,017

  31. Jatoba LC, Grossmann U, Kunze C, Ottenbacher J, Stork W (2008) Context-aware mobile health monitoring: Evaluation of different pattern recognition methods for classification of physical activity. In: 2008 30th Annual international conference of the ieee engineering in medicine and biology society, pp 5250–5253. IEEE

  32. Jian H, Chen H (2015) A portable fall detection and alerting system based on k-nn algorithm and remote medicine. China Commun 12(4):23–31

    Article  MathSciNet  Google Scholar 

  33. Kang I, Cimurs R, Lee JH, Suh IH (2020) Fusion drive: End-to-end multi modal sensor fusion for guided low-cost autonomous vehicle. In: 2020 17th International conference on ubiquitous robots (UR), pp 421–428. IEEE

  34. Kańtoch E (2017) Human activity recognition for physical rehabilitation using wearable sensors fusion and artificial neural network. In: 2017 Computing in Cardiology (CinC), pp 1–4. IEEE

  35. Klepeis NE, Nelson WC, Ott WR, Robinson JP, Tsang AM, Switzer P, Behar JV, Hern SC, Engelmann WH (2001) The national human activity pattern survey (nhaps): a resource for assessing exposure to environmental pollutants. J Expo Sci Environ Epidemiol 11(3):231–252

    Article  Google Scholar 

  36. Kose M, Incel OD, Ersoy C (2012) Online human activity recognition on smart phones. In: Workshop on mobile sensing: From smartphones and wearables to big data, vol 16, pp 11–15

  37. Kumar VS, Acharya KG, Sandeep B, Jayavignesh T, Chaturvedi A (2019) Wearable sensor-based human fall detection wireless system. In: Zungeru AM, Subashini S, Vetrivelan P (eds) Wireless communication networks and internet of things, pp 217–234. Springer

  38. Kumari P, Mathew L, Syal P (2017) Increasing trend of wearables and multimodal interface for human activity monitoring: a review. Biosens Bioelectron 90:298–307

    Article  Google Scholar 

  39. LaPier J, Chatellier M (2017) Optimizing portable pulse oximeter measurement accuracy and consistency during exercise. J Acute Care Phys Therapy 8 (3):96–105

    Article  Google Scholar 

  40. Lane ND, Miluzzo E, Lu H, Peebles D, Choudhury T, Campbell AT (2010) A survey of mobile phone sensing. IEEE Commun Magaz 48 (9):140–150

    Article  Google Scholar 

  41. Lane ND, Mohammod M, Lin M, Yang X, Lu H, Ali S, Doryab A, Berke E, Choudhury T, Campbell A (2011) Bewell: a smartphone application to monitor, model and promote wellbeing. In: 5Th International ICST conference on pervasive computing technologies for healthcare, pp 23–26

  42. Liang X, Wang G (2017) A convolutional neural network for transportation mode detection based on smartphone platform. In: 2017 IEEE 14th International conference on mobile ad hoc and sensor systems (MASS), pp 338–342. IEEE

  43. Marr B (2020) The 5 Biggest Smart Home Trends In 2020. Accessed 15 Jan 2020. https://www.forbes.com/sites/bernardmarr/2020/01/13/the-5-biggest-smart-home-trends-in-2020/#119ca3f0389b

  44. Mirri S, Prandi C, Roccetti M, Salomoni P (2017) Walking under a different sky: Urban colored routes for creative engagement and pleasure. Int J Human–Comput Interaction 33(12):1010–1021

    Article  Google Scholar 

  45. Münzner S, Schmidt P, Reiss A, Hanselmann M, Stiefelhagen R, Dürichen R (2017) Cnn-based sensor fusion techniques for multimodal human activity recognition. In: Proceedings of the 2017 ACM international symposium on wearable computers, pp 158–165

  46. Niswar M, Ilham AA, Palantei E, Sadjad RS, Ahmad A, Suyuti A, Muslimin Z, Waris T, Adi PDP et al (2013) Performance evaluation of zigbee-based wireless sensor network for monitoring patients’ pulse status. In: 2013 International conference on information technology and electrical engineering (ICITEE), pp 291–294. IEEE

  47. Nweke HF, Teh YW, Al-Garadi MA, Alo UR (2018) Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: State of the art and research challenges. Expert Syst Appl 105:233–261

    Article  Google Scholar 

  48. Panwar M, Dyuthi SR, Prakash KC, Biswas D, Acharyya A, Maharatna K, Gautam A, Naik GR (2017) Cnn based approach for activity recognition using a wrist-worn accelerometer. In: 2017 39th Annual international conference of the ieee engineering in medicine and biology society (EMBC), pp 2438–2441. IEEE

  49. Park S-M, Kim J-Y, Ko K-E, Jang I-H, Sim K-B (2013) Real-time heart rate monitoring system based on ring-type pulse oximeter sensor. J Electrical Eng Technol 8(2):376–384

    Article  Google Scholar 

  50. Pires IM, Garcia NM, Pombo N, Flórez-Revuelta F, Spinsante S (2017) Pattern recognition techniques for the identification of activities of daily living using mobile device accelerometer. arXiv:1711.00096

  51. Pires IM, Marques G, Garcia NM, Flórez-Revuelta F, Canavarro Teixeira M, Zdravevski E, Spinsante S, Coimbra M (2020) Pattern recognition techniques for the identification of activities of daily living using a mobile device accelerometer. Electronics 9(3):509

    Article  Google Scholar 

  52. Pirttikangas S, Fujinami K, Nakajima T (2006) Feature selection and activity recognition from wearable sensors. In: International symposium on ubiquitious computing systems, pp 516–527. Springer

  53. Rosner DK, Saegusa H, Friedland J, Chambliss A (2015) Walking by drawing. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, pp 397–406

  54. Sahu SN, Moharana M, Prusti PC, Chakrabarty S, Khan F, Pattanayak SK (2020) Chapter 2 - real-time data analytics in healthcare using the internet of things. In: Das H, Dey N, Emilia Balas V (eds) Real-time data analytics for large scale sensor data. advances in ubiquitous sensing applications for healthcare, vol 6, pp 37–50. Academic Press. https://doi.org/10.1016/B978-0-12-818014-3.00002-4. https://www.sciencedirect.com/science/article/pii/B9780128180143000024

  55. Sano A, Phillips AJ, Amy ZY, McHill AW, Taylor S, Jaques N, Czeisler CA, Klerman EB, Picard RW (2015) Recognizing academic performance, sleep quality, stress level, and mental health using personality traits, wearable sensors and mobile phones. In: 2015 IEEE 12th International conference on wearable and implantable body sensor networks (BSN), pp 1–6. IEEE

  56. Sarin S, Mittal A, Chugh A, Srivastava S (2022) Cnn-based multimodal touchless biometric recognition system using gait and speech. J Intell & Fuzzy Syst 42(2):981–990

    Article  Google Scholar 

  57. Škraba A, Koložvari A, Kofjač D, Stojanović R, Stanovov V, Semenkin E (2016) Streaming pulse data to the cloud with bluetooth le or nodemcu esp8266. In: 2016 5th Mediterranean conference on embedded computing (MECO), pp 428–431. IEEE

  58. Saurav S, Saini R, Singh S (2022) Vision-based techniques for fall detection in 360 videos using deep learning: Dataset and baseline results. Multimed Tools Appl 81(10):14173–14216

    Article  Google Scholar 

  59. Shoaib M, Bosch S, Scholten H, Havinga PJ, Incel OD (2015) Towards detection of bad habits by fusing smartphone and smartwatch sensors. In: 2015 IEEE International conference on pervasive computing and communication workshops (PerCom Workshops), pp 591–596. IEEE

  60. Tapia EM, Intille SS, Haskell W, Larson K, Wright J, King A, Friedman R (2007) Real-time recognition of physical activities and their intensities using wireless accelerometers and a heart rate monitor. In: 2007 11th IEEE International symposium on wearable computers, pp 37–40. IEEE

  61. Tran DN, Phan DD (2016) Human activities recognition in android smartphone using support vector machine. In: 2016 7th International conference on intelligent systems, modelling and simulation (ISMS), pp 64–68. IEEE

  62. Valarmathy G, Keerthana K, Kavithapriya J, Keerthana T (2019) Human fall detection using 3-axis mems accelerometer

  63. Wibawa AD, Purnomo MH, Marzuki A, Rumpa LD (2016) Physiological pattern of human state emotion based on ecg and pulse sensor

  64. Wu Y-C, Chen P-F, Hu Z-H, Chang C-H, Lee G-C, Yu W-C (2009) A mobile health monitoring system using rfid ring-type pulse sensor. In: 2009 Eighth IEEE International conference on dependable, autonomic and secure computing, pp 317–322. IEEE

  65. Xu B (2009) Human activity recognition using body area sensor networks. PhD thesis, Citeseer

  66. Yacchirema D, de Puga JS, Palau C, Esteve M (2018) Fall detection system for elderly people using iot and big data. Procedia Comput Sci 130:603–610

    Article  Google Scholar 

  67. Yang J, Nguyen MN, San PP, Li XL, Krishnaswamy S (2015) Deep convolutional neural networks on multichannel time series for human activity recognition. In: Twenty-fourth international joint conference on artificial intelligence

  68. Zeng M, Nguyen LT, Yu B, Mengshoel OJ, Zhu J, Wu P, Zhang J (2014) Convolutional neural networks for human activity recognition using mobile sensors. In: 6th International conference on mobile computing, applications and services, pp 197–205. IEEE

  69. Zhang P, Chen X, Ma X, Wu Y, Jiang H, Fang D, Tang Z, Ma Y (2017) Smartmtra: Robust indoor trajectory tracing using smartphones. IEEE Sensors J 17(12):3613–3624

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, University of Engineering and Management, Kolkata, 700160, West Bengal, India

    Amartya Chakraborty

  2. Department of Computer Science and Engineering, Jadavpur University, Kolkata, 700032, West Bengal, India

    Nandini Mukherjee

Authors
  1. Amartya Chakraborty
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nandini Mukherjee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Amartya Chakraborty.

Ethics declarations

Conflict of Interests

The authors have no conflicts of interest to declare.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chakraborty, A., Mukherjee, N. A deep-CNN based low-cost, multi-modal sensing system for efficient walking activity identification. Multimed Tools Appl 82, 16741–16766 (2023). https://doi.org/10.1007/s11042-022-13990-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13990-x

Keywords

Search

Navigation