Abstract
Human Activity Recognition (HAR) using smartphone sensors has been identified as a significant emerging research domain. Its application areas exhibit the performance from the intelligent tailored activity monitoring. Researchers have proposed various HAR models to recognize the human activity patterns using traditional smartphone sensor data. In addition, embedding contextual information such as data availability, sensing device orientation, body part location, axis layout, and more makes a fruitful impact on the quality of activity sensor data. The research challenge occurred due to the lack of contextual information in sensor data, leading to activity patterns ambiguity. Often, the motion sensor was separately used to acquire contextual information that consumes unnecessary computational resources. In this paper, we have used activity sensor data availability as contextual information. The proposed Deep Context Model (DCM) recognizes the activity pattern in the dual context-attention mode, i.e., static and dynamic context. The proposed model consists of convolutional and recurrent networks that find the associated activity patterns in a dual context. The convolutional networks are excellent for automatic feature extraction in a static context, whereas recurrent networks are used in a dynamic context for memorizing patterns. We evaluated the performance of the proposed model using the open accessible KU-HAR dataset. The experimental outcomes revealed that DCM has achieved the F1 score of 98.96% and 99.62% for static and dynamic context, respectively. Further, the robustness and applicability of the proposed model have been gauged using the HHAR dataset.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The experimental datasets i.e. KU-HAR and HHAR are available on the https://data.mendeley.com/datasets/45f952y38r/5 and https://archive.ics.uci.edu/dataset/344/heterogeneity+activity+recognition.
References
Akbari A, Martinez J, Jafari R (2021) Facilitating human activity data annotation via context-aware change detection on smartwatches. ACM Trans Embed Comput Syst 20:10. https://doi.org/10.1145/3431503
Arshad MH, Bilal M, Gani A (2022) Human activity recognition: review, taxonomy and open challenges. Sensors 22:10. https://doi.org/10.3390/S22176463
Baloch Z, Shaikh FK, Unar MA (2022) CNN-LSTM-based late sensor fusion for human activity recognition in big data networks. Wirel Commun Mob Comput. https://doi.org/10.1155/2022/3434100
Barua A, Fuller D, Musa S, Jiang X (2022) Exploring orientation invariant heuristic features with variant window length of 1D-CNN-LSTM in human activity recognition. Biosensors (Basel). https://doi.org/10.3390/BIOS12070549
Beaulieu A, Thullier F, Bouchard K et al (2022) Ultra-wideband data as input of a combined EfficientNet and LSTM architecture for human activity recognition. J Ambient Intell Smart Environ 14:157–172. https://doi.org/10.3233/AIS-210462
Bernaś M, Płaczek B, Lewandowski M (2022) Ensemble of RNN classifiers for activity detection using a smartphone and supporting nodes. Sensors 22:9451. https://doi.org/10.3390/S22239451
Butt A, Narejo S, Anjum MR et al (2022) Fall detection using LSTM and transfer learning. Wirel Pers Commun 126:1733–1750. https://doi.org/10.1007/S11277-022-09819-3
Domingo JD, Gómez-García-Bermejo J, Zalama E (2022) Improving human activity recognition integrating LSTM with different data sources: features, object detection and skeleton tracking. IEEE Access 10:68213–68230. https://doi.org/10.1109/ACCESS.2022.3186465
Ehatisham-Ul-haq M, Murtaza F, Azam MA, Amin Y (2022) Daily living activity recognition in-the-wild: modeling and inferring activity-aware human contexts. Electronics (Switzerland). https://doi.org/10.3390/ELECTRONICS11020226
Gao Z, Liu D, Huang K, Huang Y (2019) Context-aware human activity and smartphone position-mining with motion sensors. Remote Sens (Basel). https://doi.org/10.3390/RS11212531
García-Domínguez A, Galvan-Tejada CE, Zanella-Calzada LA et al (2020) Deep artificial neural network based on environmental sound data for the generation of a children activity classification model. PeerJ Comput Sci 6:e308. https://doi.org/10.7717/PEERJ-CS.308/SUPP-1
Ige AO, Mohd Noor MH (2022) A survey on unsupervised learning for wearable sensor-based activity recognition. Appl Soft Comput 127:109363. https://doi.org/10.1016/J.ASOC.2022.109363
Islam MdM, Nooruddin S, Karray F, Muhammad G (2023) Multi-level feature fusion for multimodal human activity recognition in Internet of Healthcare Things. Information Fusion 94:17–31. https://doi.org/10.1016/J.INFFUS.2023.01.015
Ismail WN, Alsalamah HA, Hassan MM, Mohamed E (2023) AUTO-HAR: An adaptive human activity recognition framework using an automated CNN architecture design. Heliyon 9:e13636. https://doi.org/10.1016/J.HELIYON.2023.E13636
Javed AR, Faheem R, Asim M et al (2021) A smartphone sensors-based personalized human activity recognition system for sustainable smart cities. Sustain Cities Soc 71:102970. https://doi.org/10.1016/J.SCS.2021.102970
Jethanandani M, Sharma A, Perumal T, Chang JR (2020) Multi-label classification based ensemble learning for human activity recognition in smart home. Internet of Things 12:100324. https://doi.org/10.1016/J.IOT.2020.100324
Jin F, Sengupta A, Cao S (2022) mmFall: fall detection using 4-D mmWave radar and a hybrid variational RNN AutoEncoder. IEEE Trans Autom Sci Eng 19:1245–1257. https://doi.org/10.1109/TASE.2020.3042158
Khaire P, Kumar P, Imran J (2018) Combining CNN streams of RGB-D and skeletal data for human activity recognition. Pattern Recognit Lett 115:107–116. https://doi.org/10.1016/J.PATREC.2018.04.035
Khatun MA, Yousuf MA, Ahmed S et al (2022) Deep CNN-LSTM with self-attention model for human activity recognition using wearable sensor. IEEE J Transl Eng Health Med. https://doi.org/10.1109/JTEHM.2022.3177710
Kumar P, Suresh S (2022a) RecurrentHAR: a novel transfer learning-based deep learning model for sequential, complex, concurrent, interleaved, and heterogeneous type human activity recognition. In: IETE Technical Review (Institution of Electronics and Telecommunication Engineers, India). https://doi.org/10.1080/02564602.2022.2101557
Kumar P, Suresh S (2022b) DeepTransHHAR: inter-subjects heterogeneous activity recognition approach in the non-identical environment using wearable sensors. Natl Acad Sci Lett 45:317–323. https://doi.org/10.1007/s40009-022-01126-6
Kumar P, Suresh S (2023) DeepTransHAR: a novel clustering-based transfer learning approach for recognizing the cross-domain human activities using GRUs (Gated Recurrent Units) Networks. Internet of Things (Netherlands). https://doi.org/10.1016/j.iot.2023.100681
Lima WS, Souto E, El-Khatib K et al (2019) Human activity recognition using inertial sensors in a smartphone: An overview. Sensors (Switzerland) 19:14–16. https://doi.org/10.3390/s19143213
Liu L, He J, Ren K et al (2021) An information gain-based model and an attention-based RNN for wearable human activity recognition. Entropy 23:1635. https://doi.org/10.3390/E23121635
Mekruksavanich S, Jitpattanakul A, Mekruksavanich S, Jitpattanakul A (2022) RNN-based deep learning for physical activity recognition using smartwatch sensors: a case study of simple and complex activity recognition. Math Biosci Eng 19(6):5671–5698. https://doi.org/10.3934/MBE.2022265
Mishra SR, Mishra TK, Sanyal G et al (2020) Real time human action recognition using triggered frame extraction and a typical CNN heuristic. Pattern Recognit Lett 135:329–336. https://doi.org/10.1016/J.PATREC.2020.04.031
Niemann F, Lüdtke S, Bartelt C, ten Hompel M (2022) Context-aware human activity recognition in industrial processes. Sensors. https://doi.org/10.3390/S22010134
Omolaja A, Otebolaku A, Alfoudi A (2022a) Context-aware complex human activity recognition using hybrid deep learning models. Appl Sci (Switzerland). https://doi.org/10.3390/APP12189305
Park H, Kim N, Lee GH, Choi JK (2023) MultiCNN-FilterLSTM: Resource-efficient sensor-based human activity recognition in IoT applications. Futur Gener Comput Syst 139:196–209. https://doi.org/10.1016/J.FUTURE.2022.09.024
Qu Y, Tang Y, Yang X et al (2023) Context-aware mutual learning for semi-supervised human activity recognition using wearable sensors. Expert Syst Appl. https://doi.org/10.1016/j.eswa.2023.119679
Raziani S, Azimbagirad M (2022) Deep CNN hyperparameter optimization algorithms for sensor-based human activity recognition. Neurosci Inform 2:100078. https://doi.org/10.1016/J.NEURI.2022.100078
Saguna S, Zaslavsky A, Chakraborty D (2013) Complex activity recognition using context-driven activity theory and activity signatures. ACM Trans Comput-Hum Interact. https://doi.org/10.1145/2490832
Sahoo KK, Ghosh R, Mallik S et al (2023) Wrapper-based deep feature optimization for activity recognition in the wearable sensor networks of healthcare systems. Sci Rep. https://doi.org/10.1038/s41598-022-27192-w
Sardar AW, Ullah F, Bacha J et al (2022) Mobile sensors based platform of Human Physical Activities Recognition for COVID-19 spread minimization. Comput Biol Med 146:105662. https://doi.org/10.1016/J.COMPBIOMED.2022.105662
Sena J, Barreto J, Caetano C et al (2021) Human activity recognition based on smartphone and wearable sensors using multiscale DCNN ensemble. Neurocomputing 444:226–243. https://doi.org/10.1016/J.NEUCOM.2020.04.151
Siirtola P, Röning J (2021) Context-aware incremental learning-based method for personalized human activity recognition. J Ambient Intell Humaniz Comput 12:10499–10513. https://doi.org/10.1007/S12652-020-02808-Z
Sikder N, Nahid A et al (2021) KU-HAR: An open dataset for heterogeneous human activity recognition. Pattern Recognit Lett 146:46–54. https://doi.org/10.1016/j.patrec.2021.02.024
Song L, Yu G, Yuan J, Liu Z (2021) Human pose estimation and its application to action recognition: a survey. J vis Commun Image Represent 76:103055. https://doi.org/10.1016/J.JVCIR.2021.103055
Stisen A, Blunck H, Bhattacharya S, et al (2015) Smart devices are different: Assessing and mitigating mobile sensing heterogeneities for activity recognition. In: SenSys 2015 - Proceedings of the 13th ACM Conference on embedded networked sensor systems pp 127–140. https://doi.org/10.1145/2809695.2809718
Tang HY, Tan SH, Su TY et al (2021) Upper body posture recognition using inertial sensors and recurrent neural networks. Appl Sci 11:12101. https://doi.org/10.3390/APP112412101
Tarafdar P, Bose I (2021) Recognition of human activities for wellness management using a smartphone and a smartwatch: a boosting approach. Decis Support Syst 140:113426. https://doi.org/10.1016/J.DSS.2020.113426
Ullah A, Muhammad K, Ding W et al (2021) Efficient activity recognition using lightweight CNN and DS-GRU network for surveillance applications. Appl Soft Comput 103:107102. https://doi.org/10.1016/J.ASOC.2021.107102
Wan S, Qi L, Xu X et al (2020) Deep learning models for real-time human activity recognition with smartphones. Mob Netw Appl 25:743–755. https://doi.org/10.1007/S11036-019-01445-X/FIGURES/7
Yadav SK, Tiwari K, Pandey HM, Akbar SA (2022) Skeleton-based human activity recognition using ConvLSTM and guided feature learning. Soft Comput 26:877–890. https://doi.org/10.1007/S00500-021-06238-7
Yang SH, Baek DG, Thapa K (2022) Semi-supervised adversarial learning using LSTM for human activity recognition. Sensors 22:4755. https://doi.org/10.3390/S22134755
Acknowledgements
The authors appreciate the funding from the UGC, New Delhi through the JRF, and Banaras Hindu University through the Institute of Eminence (IoE) Seed Grant. The professors and researchers in the Department of Computer Science at Banaras Hindu University are also acknowledged by the authors for their continuous and beneficial discussions on this research topic.
Author information
Authors and Affiliations
Contributions
Although Prabhat Kumar designed the architecture, conducted experiments analyzed the data, and wrote the article, S. Suresh oversaw and evaluated its writing. All of the authors have read and approved the final article.
Corresponding author
Ethics declarations
Conflict of interest
No known financial or personal conflicts of interest exist amongst the authors, they declare, which could have affected the work performed for this publication.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Kumar, P., Suresh, S. Deep Context Model (DCM): dual context-attention aware model for recognizing the heterogeneous human activities using smartphone sensors. Evolving Systems (2024). https://doi.org/10.1007/s12530-024-09570-z
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s12530-024-09570-z