Abstract
Human activity recognition is concerned with detecting different types of human movements and actions using data gathered from various types of sensors. Deep learning approaches, when applied on time series data, offer promising results over intensive handcrafted feature extraction techniques that are highly reliant on the quality of defined domain parameters. In this paper, we investigate the benefits of time series data augmentation in improving the accuracy of several deep learning models on human activity data gathered from mobile phone accelerometers. More specifically, we compare the performance of the Vanilla, Long-Short Term Memory, and Gated Recurrent Units neural network models on three open-source datasets. We use two time series data augmentation techniques and study their impact on the accuracy of the target models. The experiments show that using gated recurrent units achieves the best results in terms of accuracy and training time followed by the long-short term memory technique. Furthermore, the results show that using data augmentation significantly enhances recognition quality.
Similar content being viewed by others
References
Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Kudlur M (2016) Tensorflow: A system for large-scale machine learning. In: 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pp 265–283
Alsarhan T, Alawneh L, Al-Zinati M, Al-Ayyoub M (2019) Bidirectional gated recurrent units for human activity recognition using accelerometer data. In: 2019 IEEE SENSORS, IEEE, pp 1–4
Anguita D, Ghio A, Oneto L, Parra X, Reyes-Ortiz JL (2013) A public domain dataset for human activity recognition using smartphones. In: Esann, p 3
Behera A, Hogg DC, Cohn AG (2012) Egocentric activity monitoring and recovery. Asian conference on computer vision. Springer, Berlin, pp 519–532
Bidargaddi N, Sarela A, Klingbeil L, Karunanithi M (2007) Detecting walking activity in cardiac rehabilitation by using accelerometer. In: 2007 3rd International Conference on Intelligent Sensors, Sensor Networks and Information. IEEE, pp 555–560
Che Z, Purushotham S, Cho K, Sontag D, Liu Y (2018) Recurrent neural networks for multivariate time series with missing values. Sci Rep 8(1):1–12
Chen Y, Zhong K, Zhang J, Sun Q, Zhao X (2016) Lstm networks for mobile human activity recognition. 2016 International conference on artificial intelligence: technologies and applications. Atlantis Press, Paris
Chen Z, Zhu Q, Soh YC, Zhang L (2017) Robust human activity recognition using smartphone sensors via CT-PCA and online SVM. IEEE Trans Industr Inf 13(6):3070–3080
Cho H, Yoon SM (2018) Divide and conquer-based 1D CNN human activity recognition using test data sharpening. Sensors 18(4):1055
Cho K, Van Merriënboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Conference on empirical methods in natural language processing (EMNLP 2014), Doha, Qatar, pp 1724–1734
Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. In: NIPS 2014 workshop on deep learning
Deng J, Guo J, Xue N, Zafeiriou S (2019) Arcface: additive angular margin loss for deep face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4690–4699
Dobbin KK, Simon RM (2011) Optimally splitting cases for training and testing high dimensional classifiers. BMC Med Genom 4(1):31
Ferrari A, Micucci D, Mobilio M, Napoletano P (2020) On the personalization of classification models for human activity recognition. IEEE Access 8:32066–32079
Foerster F, Smeja M, Fahrenberg J (1999) Detection of posture and motion by accelerometry: a validation study in ambulatory monitoring. Comput Hum Behav 15(5):571–583
Gao W, Zhang L, Teng Q, Wu H, Min F, He J (2020) DanHAR: dual attention network for multimodal human activity recognition using wearable sensors.
Gers FA, Schraudolph NN, Schmidhuber J (2002) Learning precise timing with LSTM recurrent networks. J Mach Learn Res 3:115–143
Graves A, Liwicki M, Fernández S, Bertolami R, Bunke H, Schmidhuber J (2008) A novel connectionist system for unconstrained handwriting recognition. IEEE Trans Pattern Anal Mach Intell 31(5):855–868
Gutchess D, Checka N, Snorrason MS (2007) Learning patterns of human activity for anomaly detection. Intelligent computing: theory and applications. International Society for Optics and Photonics, Washington, p 65600Y
Hammerla NY, Halloran S, Plötz T (2016) Deep, convolutional, and recurrent models for human activity recognition using wearables. In: Proceedings of the twenty-fifth international joint conference on artificial intelligence, pp 1533–1540
Hassan MM, Uddin MZ, Mohamed A, Almogren A (2018) A robust human activity recognition system using smartphone sensors and deep learning. Future Gener Comput Syst 81:307–313
Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780
Husken M, Stagge P (2003) Recurrent neural networks for time series classification. Neurocomputing 50:223–235
Hyndman R, Koehler AB, Ord JK, Snyder RD (2008) Forecasting with exponential smoothing: the state space approach. Springer Science and Business Media, Cham
Jalal A, Kamal S, Kim D (2014) A depth video sensor-based life-logging human activity recognition system for elderly care in smart indoor environments. Sensors 14(7):11735–11759
Jiang W, Yin Z (2015) Human activity recognition using wearable sensors by deep convolutional neural networks. In: Proceedings of the 23rd ACM international conference on Multimedia, pp 1307–1310
Johnson RA, Miller I, Freund JE (2000) Probability and statistics for engineers. Pearson Education, London, p 642
Jordao A, Kloss R, Schwartz WR (2018) Latent HyperNet: exploring the layers of convolutional neural networks. In: 2018 International Joint Conference on Neural Networks (IJCNN), IEEE, pp 1–7
Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980
Kolekar MH, Dash DP (2016) Hidden markov model based human activity recognition using shape and optical flow based features. In: 2016 IEEE Region 10 Conference (TENCON), IEEE, pp 393–397
Kwapisz JR, Weiss GM, Moore SA (2011) Activity recognition using cell phone accelerometers. ACM SIGKDD Explor Newsl 12(2):74–82
Li H, Trocan M (2019) Deep learning of smartphone sensor data for personal health assistance. Microelectron J 88:164–172
Li F, Shirahama K, Nisar MA, Köping L, Grzegorzek M (2018) Comparison of feature learning methods for human activity recognition using wearable sensors. Sensors 18(2):679
Liciotti D, Bernardini M, Romeo L, Frontoni E (2020) A sequential deep learning application for recognising human activities in smart homes. Neurocomputing 396:501–513
Lv T, Wang X, Jin L, Xiao Y, Song M (2020) Margin-based deep learning networks for human activity recognition. Sensors 20(7):1871
Mehdiyev N, Lahann J, Emrich A, Enke D, Fettke P, Loos P (2017) Time series classification using deep learning for process planning: a case from the process industry. Proced Comput Sci 114:242–249
Micucci D, Mobilio M, Napoletano P (2017) Unimib shar: A dataset for human activity recognition using acceleration data from smartphones. Appl Sci 7(10):1101
Mozer MC (1998) The neural network house: an environment hat adapts to its inhabitants. In: Proceedings of AAAI Spring Symposium of Intelligent Environments.
Mukherjee D, Mondal R, Singh PK, Sarkar R, Bhattacharjee D (2020) EnsemConvNet: a deep learning approach for human activity recognition using smartphone sensors for healthcare applications. Multimed Tools Appl 79(41):31663–31690
Murad A, Pyun JY (2017) Deep recurrent neural networks for human activity recognition. Sensors 17(11):2556
Nweke HF, Teh YW, Al-Garadi MA, Alo UR (2018) Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: state of the art and research challenges. Expert Syst Appl 105:233–261
Oniga S, Sütő J (2014) Human activity recognition using neural networks. In: Proceedings of the 2014 15th International Carpathian Control Conference (ICCC), IEEE, pp 403–406
Osmani V, Balasubramaniam S, Botvich D (2008) Human activity recognition in pervasive health-care: supporting efficient remote collaboration. J Netw Comput Appl 31(4):628–655
Paul P, George T (2015) An effective approach for human activity recognition on smartphone. In: 2015 IEEE International Conference on Engineering and Technology (ICETECH), IEEE, pp 1–3
Plötz T, Hammerla NY, Olivier P (2011) Feature learning for activity recognition in ubiquitous computing. In: 22nd international joint conference on artificial intelligence, IJCAI 2011, pp 1729–1734
Powell HC, Hanson MA, Lach J (2007) A wearable inertial sensing technology for clinical assessment of tremor. In: 2007 IEEE Biomedical Circuits and Systems Conference, IEEE, pp 9–12
Ramasamy SR, Roy N (2018) Recent trends in machine learning for human activity recognition—a survey. Wiley Interdiscip Rev 8(4):e1254
Ravi D, Wong C, Lo B, Yang GZ (2016) A deep learning approach to on-node sensor data analytics for mobile or wearable devices. IEEE J Biomed Health Inform 21(1):56–64
Ravuri S, Stolcke A (2015) Recurrent neural network and LSTM models for lexical utterance classification. In: Sixteenth Annual Conference of the International Speech Communication Association.
Ronao CA, Cho SB (2016) Human activity recognition with smartphone sensors using deep learning neural networks. Expert Syst Appl 59:235–244
Sabatini AM, Martelloni C, Scapellato S, Cavallo F (2005) Assessment of walking features from foot inertial sensing. IEEE Trans Biomed Eng 52(3):486–494
Schuster M, Paliwal KK (1997) Bidirectional recurrent neural networks. IEEE Trans Signal Process 45(11):2673–2681
Sefen B, Baumbach S, Dengel A, Abdennadher S (2016) Human activity recognition. In: Proceedings of the 8th International Conference on Agents and Artificial Intelligence. SCITEPRESS-Science and Technology Publications, Lda, pp 488–493
Shen G, Tan Q, Zhang H, Zeng P, Xu J (2018) Deep learning with gated recurrent unit networks for financial sequence predictions. Proced Comput Sci 131:895–903
Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems. Springer, Cham, pp 568–576
Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958
Sukor AA, Zakaria A, Rahim NA (2018) Activity recognition using accelerometer sensor and machine learning classifiers. In: 2018 IEEE 14th International Colloquium on Signal Processing and its Applications (CSPA), IEEE, pp 233–238
Teng Q, Wang K, Zhang L, He J (2020) The layer-wise training convolutional neural networks using local loss for sensor-based human activity recognition. IEEE Sens J 20(13):7265–7274
Torres-Huitzil C, Alvarez-Landero A (2015) Accelerometer-based human activity recognition in smartphones for healthcare services. Mobile health. Springer, Cham, pp 147–169
Uddin MZ, Hassan MM, Almogren A, Zuair M, Fortino G, Torresen J (2017) A facial expression recognition system using robust face features from depth videos and deep learning. Comput Electr Eng 63:114–125
Uddin MZ, Hassan MM, Alsanad A, Savaglio C (2020) A body sensor data fusion and deep recurrent neural network-based behavior recognition approach for robust healthcare. Inform Fus 55:105–115
Ullah M, Ullah H, Khan SD, Cheikh FA (2019) Stacked Lstm network for human activity recognition using smartphone data. In: 2019 8th European Workshop on Visual Information Processing (EUVIP), IEEE, pp 175–180
Veeriah V, Zhuang N, Qi GJ (2015) Differential recurrent neural networks for action recognition. In: Proceedings of the IEEE international conference on computer vision, pp 4041–4049
Vepakomma P, De D, Das SK, Bhansali S (2015) A-Wristocracy: Deep learning on wrist-worn sensing for recognition of user complex activities. In: 2015 IEEE 12th International conference on wearable and implantable body sensor networks (BSN), IEEE, pp 1–6
Vu TH, Dang A, Dung L, Wang JC (2017) Self-gated recurrent neural networks for human activity recognition on wearable devices. In: Proceedings of the on Thematic Workshops of ACM Multimedia 2017, pp 179–185
Wang J, Chen Y, Hao S, Peng X, Hu L (2019) Deep learning for sensor-based activity recognition: a survey. Pattern Recogn Lett 119:3–11
Woodruff RB, Gardial SF (1996) Know your customer: new approaches to customer value and satisfaction blackwell. Cambridge University, Cambridge
Woznowski P, King R, Harwin W, Craddock I (2016) A human activity recognition framework for healthcare applications: ontology, labelling strategies, and best practice. In: IoTBD, pp 369–377
Wu GE, Xue S (2008) Portable preimpact fall detector with inertial sensors. IEEE Trans Neural Syst Rehabil Eng 16(2):178–183
Xia K, Huang J, Wang H (2020) LSTM-CNN architecture for human activity recognition. IEEE Access 8:56855–56866
Yager RR (2008) Time series smoothing and OWA aggregation. IEEE Trans Fuzzy Syst 16(4):994–1007
Zahin A, Hu RQ (2019) Sensor-based human activity recognition for smart healthcare: a semi-supervised machine learning. International conference on artificial intelligence for communications and networks. Springer, Cham, pp 450–472
Zainudin MS, Sulaiman MN, Mustapha N, Perumal T (2015) Activity recognition based on accelerometer sensor using combinational classifiers. In: 2015 IEEE Conference on Open Systems (Icos), IEEE, pp 68–73
Zeng M, Nguyen LT, Yu B, Mengshoel OJ, Zhu J, Wu P, Zhang J (2014) Convolutional neural networks for human activity recognition using mobile sensors. In: 6th International Conference on Mobile Computing, Applications and Services, IEEE, pp 197–205
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Alawneh, L., Alsarhan, T., Al-Zinati, M. et al. Enhancing human activity recognition using deep learning and time series augmented data. J Ambient Intell Human Comput 12, 10565–10580 (2021). https://doi.org/10.1007/s12652-020-02865-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-020-02865-4