Human Activity Recognition Based on Wearable Sensor Using Hierarchical Deep LSTM Networks

  • LuKun WangEmail author
  • RuYue Liu


In recent years, with the rapid development of artificial intelligence, human activity recognition has become a research focus. The complex, dynamic and variable features of human activities lead to the relatively low accuracy of the traditional recognition algorithms. In order to solve the problem, this paper will propose a novel structure named hierarchical deep LSTM (H-LSTM) based on long short-term memory. Firstly, the original sensor data are preprocessed by smoothing and denoising; then, the feature will be selected and extracted by time–frequency-domain method. Secondly, H-LSTM is applied to the classification of these activities. Three public UCI datasets are used to conduct simulation with the realization of the automatic extraction of feature vectors and classification of outputting recognition results. Finally, the simulation results testify to the outperformance of the H-LSTM network over other deep learning algorithms. The accuracy of H-LSTM network in human activity recognition is proved to be 99.15%.


Human activity recognition Acceleration sensor Recurrent neural network (RNN) Long short-term memory (LSTM) 



This work is supported by the National Natural Science Foundation of Shandong Province (ZR2018BF005), the Scientific Research Foundation of Shandong University of Science and Technology for Recruited Talents (2017RCJJ077), the Shandong Province Higher Educational Science and Technology Program (J17KB167) and the Science and Technology Program of Taian (2017GX0014, 2018ZC0284).


  1. 1.
    S. Ali, N.A. Khan, M. Haneef, X.J.C. Luo, Blind source separation schemes for mono-sensor and multi-sensor systems with application to signal detection. Circuits Syst. Signal Process. 36(11), 4615–4636 (2017)MathSciNetCrossRefGoogle Scholar
  2. 2.
    K. Altun, B. Barshan, O. Tunçel, Comparative study on classifying human activities with miniature inertial and magnetic sensors. Pattern Recogn. 43(10), 3605–3620 (2010)CrossRefzbMATHGoogle Scholar
  3. 3.
    D. Anguita, A. Ghio, L. Oneto, X. Parra, J.L. Reyes-Ortiz, Human activity recognition on smartphones using a multiclass hardware-friendly support vector machine, in International Workshop on Ambient Assisted Living (Springer, Berlin, 2012), pp. 216–223Google Scholar
  4. 4.
    J. Azorin-Lopez, M. Saval-Calvo, A. Fuster-Guillo, J. Garcia-Rodriguez, A novel prediction method for early recognition of global human behaviour in image sequences. Neural Process. Lett. 43(2), 363–387 (2015)CrossRefGoogle Scholar
  5. 5.
    M. Babiker, O.O. Khalifa, K.K. Htike, A. Hassan, M. Zaharadeen, Automated daily human activity recognition for video surveillance using neural network, in Proceedings of 2017 4th IEEE International Conference on Smart Instrumentation, Measurement and Application (IEEE, 2017), pp. 1–5Google Scholar
  6. 6.
    O. Banos, J.-M. Galvez, M. Damas, A. Guillen, L.-J. Herrera, H. Pomares, I. Rojas, C. Villalonga, C.S. Hong, S. Lee, Multiwindow fusion for wearable activity recognition, in International Work-Conference on Artificial Neural Networks (Springer, Berlin, 2015), pp. 290–297Google Scholar
  7. 7.
    A. Bulling, U. Blanke, B. Schiele, A tutorial on human activity recognition using body-worn inertial sensors. ACM Comput. Surv. 46(3), 33 (2014)CrossRefGoogle Scholar
  8. 8.
    P. Casale, O. Pujol, P. Radeva, Human activity recognition from accelerometer data using a wearable device, in Proceedings of Iberian Conference on Pattern Recognition and Image Analysis (Springer, Berlin, 2011), pp. 289–296Google Scholar
  9. 9.
    L. Chen, J. Hoey, C.D. Nugent, D.J. Cook, Z. Yu, Sensor-based activity recognition. IEEE Trans. Syst. Man, Cybern. C, Appl. Rev. 42(6), 790–808 (2012)CrossRefGoogle Scholar
  10. 10.
    L. Chen, H. Wei, J. Ferryman, ReadingAct RGB-D action dataset and human action recognition from local features. Pattern Recogn. Lett. 50, 159–169 (2014)CrossRefGoogle Scholar
  11. 11.
    W.-H. Chen, C.A.B. Baca, C.-H. Tou, LSTM-RNNs combined with scene information for human activity recognition, in 2017 IEEE 19th International Conference on e-Health Networking, Applications and Services (2017), pp. 1–6Google Scholar
  12. 12.
    M. Ciman, M. Donini, O. Gaggi, F. Aiolli, Stairstep recognition and counting in a serious game for increasing users’ physical activity. Pers. Ubiquitous Comput. 20(6), 1015–1033 (2016)CrossRefGoogle Scholar
  13. 13.
    K. Cui, X. Jing, Research on prediction model of geotechnical parameters based on BP neural network. Neural. Comput. Appl. (2018). Google Scholar
  14. 14.
    M. Edel, E. Köppe, Binarized-blstm-rnn based human activity recognition, in Proceedings of 2016 International Conference on Indoor Positioning and Indoor Navigation (IEEE, 2016), pp. 1–7Google Scholar
  15. 15.
    P. Esfahani, H.T. Malazi, PAMS: a new position-aware multi-sensor dataset for human activity recognition using smartphones, in Proceedings of 2017 19th IEEE International Symposium on Computer Architecture and Digital Systems (IEEE, 2017), pp. 1–7Google Scholar
  16. 16.
    L. Fan, Z. Wang, H. Wang, Human activity recognition model based on decision tree, in Proceedings of 2013 International Conference on Advanced Cloud and Big Data (IEEE, 2013), pp. 64–68Google Scholar
  17. 17.
    F. Gu, K. Khoshelham, S. Valaee, J. Shang, R. Zhang, Locomotion activity recognition using stacked denoising autoencoders. IEEE Internet Things. 5(3), 2085–2093 (2018)CrossRefGoogle Scholar
  18. 18.
    P. Gupta, T. Dallas, Feature selection and activity recognition system using a single triaxial accelerometer. IEEE Trans. Biomed. Eng. 61(6), 1780–1786 (2014)CrossRefGoogle Scholar
  19. 19.
    H.P. Gupta, H.S. Chudgar, S. Mukherjee, T. Dutta, K. Sharma, A continuous hand gestures recognition technique for human–machine interaction using accelerometer and gyroscope sensors. IEEE Sens. J. 16(16), 6425–6432 (2016)CrossRefGoogle Scholar
  20. 20.
    S. Hochreiter, J. Schmidhuber, LSTM can solve hard long time lag problems, in Advances in Neural Information Processing Systems (1997), pp. 473–479Google Scholar
  21. 21.
    A. Jain, V. Kanhangad, Investigating gender recognition in smart-phones using accelerometer and gyroscope sensor readings, in Proceedings of 2016 International Conference on Computational Techniques in Information and Communication Technologies (2016), pp. 597–602Google Scholar
  22. 22.
    D. Li, H. Zhang, M.J.C. Zhang, Wavelet de-noising and genetic algorithm-based least squares twin SVM for classification of arrhythmias. Circuits Syst. Signal Process. 36(7), 2828–2846 (2017)MathSciNetCrossRefGoogle Scholar
  23. 23.
    C.L. Liu, C.H. Lee, P.M. Lin, A fall detection system using k-nearest neighbor classifier. Expert Syst. Appl. 37(10), 7174–7181 (2010)CrossRefGoogle Scholar
  24. 24.
    B. Long, M. Li, H. Wang, S.J.C. Tian, Diagnostics of analog circuits based on LS-SVM using time-domain features. Circuits Syst. Signal Process. 32(6), 2683–2706 (2013)MathSciNetCrossRefGoogle Scholar
  25. 25.
    Y. Lu, Y. Wei, L. Liu, J. Zhong, L. Sun, Y. Liu, Towards unsupervised physical activity recognition using smartphone accelerometers. Multimed. Tools Appl. 76(8), 10701–10719 (2017)CrossRefGoogle Scholar
  26. 26.
    D. Micucci, M. Mobilio, P. Napoletano, Unimib shar: a dataset for human activity recognition using acceleration data from smartphones. Appl. Sci. 7(10), 1101 (2017)CrossRefGoogle Scholar
  27. 27.
    M. Milenkoski, K. Trivodaliev, S. Kalajdziski, M. Jovanov, B.R. Stojkoska, Real time human activity recognition on smartphones using LSTM Networks, in Proceedings of 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (IEEE, 2018), pp. 1126–1131Google Scholar
  28. 28.
    U.M. Nunes, D.R. Faria, P. Peixoto, A human activity recognition framework using max-min features and key poses with differential evolution random forests classifier. Pattern Recogn. Lett. 99, 21–31 (2017)CrossRefGoogle Scholar
  29. 29.
    F. Ordóñez, D. Roggen, Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors. 16(1), 115 (2016)CrossRefGoogle Scholar
  30. 30.
    J. Park, K. Jang, S.-B. Yang, Deep neural networks for activity recognition with multi-sensor data in a smart home, in 2018 4th IEEE World Forum on Internet of Things (IEEE, 2018), pp. 155–160Google Scholar
  31. 31.
    S.U. Park, J.H. Park, M.A. Al-masni, M.A. Al-antari, M.Z. Uddin, T.S. Kim, A depth camera-based human activity recognition via deep learning recurrent neural network for health and social care services. Proced. Comput. Sci. 100, 78–84 (2016)CrossRefGoogle Scholar
  32. 32.
    S.Y. Park, H. Ju, C.G. Park, Stance phase detection of multiple actions for military drill using foot-mounted IMU. Sensors. 14, 16 (2016)Google Scholar
  33. 33.
    J.S. Peng, Y.M. Shao, Intelligent method for identifying driving risk based on V2V multisource big data. Complexity. 2018(1), 1–9 (2018)zbMATHGoogle Scholar
  34. 34.
    S.J. Preece, J.Y. Goulermas, L.P. Kenney, D. Howard, A comparison of feature extraction methods for the classification of dynamic activities from accelerometer data. IEEE Trans. Biomed. Eng. 56(3), 871–879 (2009)CrossRefGoogle Scholar
  35. 35.
    J.L. Reyes-Ortiz, L. Oneto, A. Ghio, A. Samá, D. Anguita, X. Parra, Human activity recognition on smartphones with awareness of basic activities and postural transitions, in Proceedings of International Conference on Artificial Neural Networks (Springer, Berlin, 2014), pp. 177–184Google Scholar
  36. 36.
    G. Sridevi, S.J.C. Srinivas Kumar, Image inpainting based on fractional-order nonlinear diffusion for image reconstruction. Circuits Syst. Signal Process (2019). Google Scholar
  37. 37.
    A. Subasi, M. Radhwan, R. Kurdi, K. Khateeb, IoT based mobile healthcare system for human activity recognition, in Proceedings of 2018 15th IEEE Conference on Learning and Technology (IEEE, 2018), pp. 29–34Google Scholar
  38. 38.
    P. Susarla, U. Agrawal, D.B. Jayagopi, Human weapon-activity recognition in surveillance videos using structural-RNN, in Proceedings of 2nd Mediterranean Conference on Pattern Recognition and Artificial Intelligence (ACM, New York, 2018), pp. 101–107Google Scholar
  39. 39.
    G. Vavoulas, M. Pediaditis, E.G. Spanakis, M. Tsiknakis, The MobiFall dataset: An initial evaluation of fall detection algorithms using smartphones, in Proceedings of 13th IEEE International Conference on BioInformatics and BioEngineering (2013), pp. 1–4Google Scholar
  40. 40.
    T.H. Vu, A. Dang, L. Dung, J.-C. Wang, Self-gated recurrent neural networks for human activity recognition on wearable devices, in Proceedings of the on Thematic Workshops of ACM Multimedia 2017 (ACM, New York, 2017), pp. 179–185Google Scholar
  41. 41.
    L. Wang, Recognition of human activities using continuous autoencoders with wearable sensors. Sensors. 16(2), 189 (2016)CrossRefGoogle Scholar
  42. 42.
    L. Wang, Three-dimensional convolutional restricted Boltzmann machine for human activity recognition from RGB-D video. Eurasip J. Image Video Process. 2018(1), 120 (2018)CrossRefGoogle Scholar
  43. 43.
    J.H. Wu, W. Wei, L. Zhang et al., Risk assessment of hypertension in steel workers based on LVQ and fisher-SVM deep excavation. IEEE Access. 7, 23109–23119 (2019)CrossRefGoogle Scholar
  44. 44.
    F. Xiao, J. Chen, X.H. Xie, L. Gui, J.L. Sun, W. none Ruchuan, SEARE: a system for exercise activity recognition and quality evaluation based on green sensing. IEEE Trans. Emerg. Top. Comput. (2018). Google Scholar
  45. 45.
    A. Yassine, S. Singh, A.J.I.A. Alamri, Mining human activity patterns from smart home big data for health care applications. IEEE Access. 5, 13131–13141 (2017)CrossRefGoogle Scholar
  46. 46.
    H. Zhang, A.C. Berg, M. Maire, J. Malik, SVM-KNN: Discriminative nearest neighbor classification for visual category recognition, in Proceedings of 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06) (IEEE, 2006), pp. 2126–2136Google Scholar
  47. 47.
    X. Zhang, Y. Li, R. Kotagiri, L. Wu, Z. Tari, M. Cheriet, KRNN: k rare-class nearest neighbour classification. Pattern Recogn. 62, 33–44 (2017)CrossRefGoogle Scholar
  48. 48.
    X.Y. Zhang, F. Yin, Y.M. Zhang, C.L. Liu, Y. Bengio, Drawing and recognizing Chinese characters with recurrent neural network. IEEE Trans. Pattern. Anal. 40(4), 849–862 (2018)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Information EngineeringShandong University of Science and TechnologyTaianChina
  2. 2.College of Computer Science and EngineeringShandong University of Science and TechnologyQingdaoChina

Personalised recommendations