Human Activity Recognition (HAR) has attracted much attention from researchers in the recent past. The intensification of research into HAR lies in the motive to understand human behaviour and inherently anticipate human intentions. Human activity data obtained via wearable sensors like gyroscope and accelerometer is in the form of time series data, as each reading has a timestamp associated with it. For HAR, it is important to extract the relevant temporal features from raw sensor data. Most of the approaches for HAR involves a good amount of feature engineering and data pre-processing, which in turn requires domain expertise. Such approaches are time-consuming and are application-specific. In this work, a Deep Neural Network based model, which uses Convolutional Neural Network, and Gated Recurrent Unit is proposed as an end-to-end model performing automatic feature extraction and classification of the activities as well. The experiments in this work were carried out using the raw data obtained from wearable sensors with nominal pre-processing and don’t involve any handcrafted feature extraction techniques. The accuracies obtained on UCI-HAR, WISDM, and PAMAP2 datasets are 96.20%, 97.21%, and 95.27% respectively. The results of the experiments establish that the proposed model achieved superior classification performance than other similar architectures.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Khan ZA, Sohn W (2013) A hierarchical abnormal human activity recognition system based on R-transform and kernel discriminant analysis for elderly health care. Computing 95:109–127. https://doi.org/10.1007/s00607-012-0216-x
Cornacchia M, Ozcan K, Zheng Y, Velipasalar S (2016) A survey on activity detection and classification using wearable sensors. IEEE Sens J 17:386–403
Anguita D, Ghio A, Oneto L, Parra X, Reyes-Ortiz JL (2013) A public domain dataset for human activity recognition using smartphones. In: ESANN
Feng Z, Mo L, Li M (2015) A Random Forest-based ensemble method for activity recognition. In: 37th Annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 5074–5077
Jain A, Kanhangad V (2017) Human activity classification in smartphones using accelerometer and gyroscope sensors. IEEE Sens J 18(3):1169–1177
Wang Z, Wu D, Chen J, Ghoneim A, Hossain MA (2016) A triaxial accelerometer-based human activity recognition via EEMD-based features and game-theory-based feature selection. IEEE Sens J. 16(9):3198–3207
Guo M, Wang Z, Yang N, Li Z, An T (2018) A multisensor multiclassifier hierarchical fusion model based on entropy weight for human activity recognition using wearable inertial sensors. IEEE Trans Hum–Mach Syst. 49(1):105–111
Nweke HF, Teh YW, Al-Garadi MA, Alo UR (2018) Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: State of the art and research challenges. Expert Syst Appl 105:233–261
Zhang X, Zhang Y, Hu Q (2019) Deep learning based vein segmentation from susceptibility-weighted images. Computing 101(6):637–652. https://doi.org/10.1007/s00607-018-0677-7
Yu X, Dong H (2018) PTL-CFS based deep convolutional neural network model for remote sensing classification. Computing 100(8):773–785. https://doi.org/10.1007/s00607-018-0609-6
Semwal VB, Mondal K, Nandi GC (2017) Robust and accurate feature selection for humanoid push recovery and classification: deep learning approach. Neural Comput Appl 28:565–574
Ren S, He K, Girshick R, Sun J(2015) Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint http://arxiv.org/abs/1506.01497
Al-Makhadmeh Z, Tolba A (2020) Automatic hate speech detection using killer natural language processing optimizing ensemble deep learning approach. Computing 102(2):501–522. https://doi.org/10.1007/s00607-019-00745-0
Le Cun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD (1989) Handwritten digit recognition with a back-propagation network. In: Proceedings of the 2nd international conference on neural information processing systems, pp 396–404
Lee H, Grosse R, Ranganath R, Ng AY (2009) Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In: Proceedings of the 26th annual international conference on machine learning, pp 609–616
Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint http://arxiv.org/abs/1412.3555
Wan S, Qi L, Xu X, Tong C, Gu Z (2020) Deep learning models for real-time human activity recognition with smartphones. Mobile Networks Appl 25(2):743–755
Liu CL, Hsaio WH, Tu YC (2018) Time series classification with multivariate convolutional neural network. IEEE Trans Ind Electron 66(6):4788–4797
Cui Z, Chen W, Chen Y (2016) Multi-scale convolutional neural networks for time series classification. arXiv preprint http://arxiv.org/abs/1603.06995
Ignatov A (2018) Real-time human activity recognition from accelerometer data using Convolutional Neural Networks. Appl Soft Comput 62:915–922
Ronao CA, Cho SB (2016) Human activity recognition with smartphone sensors using deep learning neural networks. Expert Syst Appl 59:235–244
Chen Y, Zhong K, Zhang J, Sun Q, Zhao X (2016) LSTM networks for mobile human activity recognition. In: 2016 International conference on artificial intelligence: technologies and applications pp 50–53
Kwapisz JR, Weiss GM, Moore SA (2011) Activity recognition using cell phone accelerometers. ACM SIGKDD Explor Newsl 12(2):74–82
Ullah M, Ullah H, Khan SD, Cheikh FA (2019) Stacked lstm network for human activity recognition using smartphone data. In: 8th European workshop on visual information processing (EUVIP). IEEE, pp 175–180
Yu S, Qin L (2018) Human activity recognition with smartphone inertial sensors using bidir-lstm networks. In: 3rd international conference on mechanical, control and computer engineering (ICMCCE). IEEE, pp 219–224
Zhao Y, Yang R, Chevalier G, Xu X, Zhang Z (2018) Deep residual bidir-LSTM for human activity recognition using wearable sensors. Math Problems Eng. https://doi.org/10.1155/2018/7316954
Xu C, Chai D, He J, Zhang X, Duan S (2019) InnoHAR: A deep neural network for complex human activity recognition. IEEE Access 7:9893–9902
Mutegeki R, Han DS (2020) A CNN-LSTM approach to human activity recognition. In: International conference on artificial intelligence in information and communication (ICAIIC). IEEE, pp 362–366
Xia K, Huang J, Wang H (2020) LSTM-CNN architecture for human activity recognition. IEEE Access 8:56855–56866
Canizo M, Triguero I, Conde A, Onieva E (2019) Multi-head CNN–RNN for multi-time series anomaly detection: An industrial case study. Neurocomputing 363:246–260
Karim F, Majumdar S, Darabi H, Chen S (2017) LSTM fully convolutional networks for time series classification. IEEE access 6:1662–1669
Kaushik S, Choudhury A, Dasgupta N, Natarajan S, Pickett LA, Dutt V (2020) Ensemble of multi-headed machine learning architectures for time-series forecasting of healthcare expenditures. Applications of Machine Learning. Springer, Singapore, pp 199–216
Patil P, Kumar KS, Gaud N, Semwal VB (2019) Clinical human gait classification: extreme learning machine approach. In: 1st International conference on advances in science, engineering and robotics technology (ICASERT). IEEE, pp 1–6
Semwal VB, Gaud N, Nandi GC (2019) Human gait state prediction using cellular automata and classification using ELM. Machine intelligence and signal analysis. Springer, Singapore, pp 135–145
Wang Z, Wu D, Gravina R, Fortino G, Jiang Y, Tang K (2017) Kernel fusion based extreme learning machine for cross-location activity recognition. Information Fusion 37:1–9
Quaid MA, Jalal A (2020) Wearable sensors based human behavioral pattern recognition using statistical features and reweighted genetic algorithm. Multimed Tools Appl 79(9):6061–6083
Zhang Y, Zhang Z, Zhang Y, Bao J, Zhang Y, Deng H (2019) Human activity recognition based on motion sensor using u-net. IEEE Access 7:75213–75226
Lu J, Zheng X, Sheng M, Jin J, Yu S (2020) Efficient human activity recognition using a single wearable sensor. IEEE Internet Things J 7(11):11137–11146
Reiss A, Stricker D (2012) Introducing a new benchmarked dataset for activity monitoring. In: 16th International symposium on wearable computers. IEEE, pp 108–109
Ignatov AD, Strijov VV (2016) Human activity recognition using quasiperiodic time series collected from a single tri-axial accelerometer. Multimed Tools Appl 75(12):7257–7270
Sadouk L. CNN approaches for time series classification (2019) In: Time series analysis-data, methods, and applications. IntechOpen, pp 1–23
Semwal VB, Nandi GC (2016) Generation of joint trajectories using hybrid automate-based model: a rocking block-based approach. IEEE Sens J 16(14):5805–5816
Gupta A, Semwal VB (2020) Multiple task human gait analysis and identification: ensemble learning approach. Emotion and Information Processing. Springer, Cham, pp 185–197
Raj M, Semwal VB, Nandi GC (2018) Bidirectional association of joint angle trajectories for humanoid locomotion: the restricted Boltzmann machine approach. Neural Comput Appl 30(6):1747–1755
Nandi GC, Semwal VB, Raj M, Jindal A (2016) Modeling bipedal locomotion trajectories using hybrid automata. In: IEEE region 10 conference (TENCON). IEEE, pp 1013–1018
LeCun Y, Bengio Y (1995) Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks 3361(10)
Bengio Y, Simard P, Frasconi P (1994) Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Networks 5(2):157–166
Cho K, Van Merriënboer B, Bahdanau D, Bengio Y (2014) On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint http://arxiv.org/abs/1409.1259
Hammerla NY, Halloran S, Plötz T (2016) Deep, convolutional, and recurrent models for human activity recognition using wearables. arXiv preprint http://arxiv.org/abs/1604.08880
Yang Z, Raymond OI, Zhang C, Wan Y, Long J (2018) DFTerNet: Towards 2-bit dynamic fusion networks for accurate human activity recognition. IEEE Access 6:56750–56764
Cheng X, Zhang L, Tang Y, Liu Y, Wu H, He J (2020) Real-time human activity recognition using conditionally parametrized convolutions on mobile and wearable devices. arXiv preprint http://arxiv.org/abs/2006.03259
Zeng M, Gao H, Yu T, Mengshoel OJ, Langseth H, Lane I, Liu X (2018) Understanding and improving recurrent networks for human activity recognition by continuous attention. In: Proceedings of the 2018 ACM international symposium on wearable computers pp 56–63
The author(s) also like to express thanks to SERB, DST, Govt. of India for funding the project under the schema of Early Career Award (ECR), with file no DST No: ECR/2018/000203 dated 04/06/2019.
Conflict of interest
We, the authors declare that we have no conflict of interest with any person or organization. This manuscript is based on original research findings done by the authors themselves.
All the ethical issues have been taken care of while writing the manuscript and we have complied with all the standards to the best of our knowledge.
About this article
Cite this article
Dua, N., Singh, S.N. & Semwal, V.B. Multi-input CNN-GRU based human activity recognition using wearable sensors. Computing 103, 1461–1478 (2021). https://doi.org/10.1007/s00607-021-00928-8
- Deep neural networks
- Human activity recognition
- Long short term memory (LSTM)
Mathematics Subject Classification
- Primary Classification: 68T