Abstract
Human activity recognition (HAR) using an accelerometer can provide valuable information for understanding user context. Therefore, several studies have been conducted using deep learning to increase the recognition rate of activity classification. However, the existing dataset that is publicly available for HAR tasks contains limited data. Previous works have applied data augmentation methods that simply transform the entire accelerometer-signal dataset. However, the label of the augmented signal cannot be easily recognized by humans, and the augmentation methods cannot ensure that the label of the signal is preserved. Therefore, we propose a novel data augmentation method that reflects the characteristics of the sensor signal and can preserve the label of the augmented signal by generating partially occluded data of the accelerometer signals. To generate the augmented data, we apply time-warping, which deforms the time-series data in the time direction. We handle jittering effects and subsequently apply data masking to drop out a part of the input signals. We compare the performance of the proposed augmentation method with that of conventional methods by using two public datasets and an activity recognition model based on convolutional neural networks. The experimental results show that the proposed augmentation method improves the recognition rate of the activity classification model, regardless of the dataset. Additionally, the proposed method shows superior performance over conventional methods on the two datasets.
Similar content being viewed by others
References
Abu Alsheikh M, Selim A, Niyato D, Doyle L, Lin S, Tan H (2016) Deep activity recognition models with triaxial accelerometers. In: AAAI Conference on artificial intelligence, AI Access Foundation, United States, vol WS-16-01 - WS-16-15. pp 8–13
Avilés-Cruz C, Ferreyra-Ramírez A, Zúñiga-López A, Villegas-Cortéz J (2019) Coarse-fine convolutional deep-learning strategy for human activity recognition. Sensors 19(7). https://doi.org/10.3390/s19071556, https://www.mdpi.com/1424-8220/19/7/1556
Chollet F, et al. (2015) Keras: Deep learning library for theano and tensorflow. https://keras io/k 7(8):T1
Chung S, Lim J, Noh KJ, Kim G, Jeong H (2019) Sensor data acquisition and multimodal sensor fusion for human activity recognition using deep learning. Sensors 19(7). https://doi.org/10.3390/s19071716. https://www.mdpi.com/1424-8220/19/7/1716
Cui X, Goel V, Kingsbury B (2015) Data augmentation for deep neural network acoustic modeling. IEEE/ACM Trans Audio Speech Lang Process (TASLP) 23(9):1469–1477
DeVries T, Taylor GW (2017) Improved regularization of convolutional neural networks with cutout. arXiv:1708.04552
Dwibedi D, Misra I, Hebert M (2017) Cut, paste and learn: Surprisingly easy synthesis for instance detection. In: 2017 IEEE International Conference on Computer Vision (ICCV). pp 1310–1319
Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov RR (2012). Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580
Inoue M, Inoue S, Nishida T (2018) Deep recurrent neural network for mobile human activity recognition with high throughput. Artif Life Robot 23 (2):173–185. https://doi.org/10.1007/s10015-017-0422-x
Jeong CY, Kim M (2019) An energy-efficient method for human activity recognition with segment-level change detection and deep learning. Sensors 19(17). https://doi.org/10.3390/s19173688.https://www.mdpi.com/1424-8220/19/17/3688
Jiang W, Yin Z (2015) Human activity recognition using wearable sensors by deep convolutional neural networks. In: Proceedings of the 23rd ACM International conference on multimedia, ACM, New York, NY, USA, MM ’15. https://doi.org/10.1145/2733373.2806333, pp 1307–1310
Kalouris G, Zacharaki EI, Megalooikonomou V (2019) Improving cnn-based activity recognition by data augmentation and transfer learning. In: 2019 IEEE 17th International Conference on Industrial Informatics (INDIN), vol 1, pp 1387–1394
Kim M, Jeong CY (2020) Label-preserving data augmentation for mobile sensor data. Multid Syst Signal Process. https://doi.org/10.1007/s11045-020-00731-2
Kim M, Jeong CY, Shin HC (2018) Activity recognition using fully convolutional network from smartphone accelerometer. In: 2018 International conference on information and communication technology convergence (ICTC). https://doi.org/10.1109/ICTC.2018.8539419, pp 1482–1484
Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. In: Bengio Y, LeCun Y (eds) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. arXiv:1412.6980
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105
Kwapisz JR, Weiss GM, Moore SA (2011) Activity recognition using cell phone accelerometers. SIGKDD Explor Newsl 12(2):74–82. https://doi.org/10.1145/1964897.1964918
Mathur A, Zhang T, Bhattacharya S, Veličković P, Joffe L, Lane ND, Kawsar F, Lió P (2018) Using deep data augmentation training to address software and hardware heterogeneities in wearable and smartphone sensing devices. In: Proceedings of the 17th ACM/IEEE International conference on information processing in sensor networks, IEEE Press, IPSN ’18. https://doi.org/10.1109/IPSN.2018.00048, pp 200–211
Micucci D, Mobilio M, Napoletano P (2017) Unimib shar: A dataset for human activity recognition using acceleration data from smartphones. Applied Sciences 7(10). https://doi.org/10.3390/app7101101, http://www.mdpi.com/2076-3417/7/10/1101
Moya Rueda F, Grzeszick R, Fink GA, Feldhorst S, ten Hompel M (2018) Convolutional neural networks for human activity recognition using body-worn sensors. Informatics 5(2). https://doi.org/10.3390/informatics5020026. http://www.mdpi.com/2227-9709/5/2/26
Mukherjee D, Mondal R, Singh PK, Sarkar R, Bhattacharjee D (2020) Ensemconvnet: a deep learning approach for human activity recognition using smartphone sensors for healthcare applications. Multimed Tools Appl https://doi.org/10.1007/s11042-020-09537-7
Murad A, Pyun JY (2017) Deep recurrent neural networks for human activity recognition. Sensors 17(11). https://doi.org/10.3390/s17112556. http://www.mdpi.com/1424-8220/17/11/2556
Nanni L, Maguolo G, Paci M (2019) Data augmentation approaches for improving animal audio classification. arXiv:191207756
Ohashi H, Al-Naser MOA, Ahmed S, Akiyama T, Sato T, Nguyen P, Nakamura K, Dengel A (2017) Augmenting wearable sensor data with physical constraint for dnn-based human-action recognition. In: Time series workshop. Time Series Workshop @ ICML, befindet sich ICML 2017 August 11-11, Sydney, Australia
Pan Y, He F, Yu H (2020) Learning social representations with deep autoencoder for recommender system. World Wide Web 23(4):2259–2279. https://doi.org/10.1007/s11280-020-00793-z
Park DS, Chan W, Zhang Y, Chiu CC, Zoph B, Cubuk ED, Le QV (2019) Specaugment: A simple data augmentation method for automatic speech recognition. Interspeech 2019. https://doi.org/10.21437/interspeech.2019-2680
Piczak KJ (2015) Environmental sound classification with convolutional neural networks. In: 2015 IEEE 25th International workshop on machine learning for signal processing (MLSP). IEEE, pp 1–6
Quan Q (2020) He F. A multi-phase blending method with incremental intensity for training detection networks. Vis Comput, Li H. https://doi.org/10.1007/s00371-020-01796-7
Ravì D, Wong C, Lo B, Yang G (2017) A deep learning approach to on-node sensor data analytics for mobile or wearable devices. IEEE J Biomed Health Inf 21(1):56–64. https://doi.org/10.1109/JBHI.2016.2633287
Salamon J, Bello JP (2017) Deep convolutional neural networks and data augmentation for environmental sound classification. IEEE Signal Process Lett 24(3):279–283
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:14091556
Stisen A, Blunck H, Bhattacharya S, Prentow TS, Kjærgaard MB, Dey A, Sonne T, Jensen MM (2015) Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition. In: Proceedings of the 13th ACM Conference on embedded networked sensor systems, association for computing machinery, New York, NY, USA, SenSys ’15. https://doi.org/10.1145/2809695.2809718, pp 127–140
Um TT, Pfister FMJ, Pichler D, Endo S, Lang M, Hirche S, Fietzek U, Kulić D (2017) Data augmentation of wearable sensor data for parkinson’s disease monitoring using convolutional neural networks. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction, Association for Computing Machinery, New York, NY, USA, ICMI ’17. https://doi.org/10.1145/3136755.3136817, pp 216–220
Weiss G, Lockhart J (2012) The impact of personalization on smartphone-based activity recognition. In: Twenty-sixth AAAI conference on artificial intelligence. https://www.aaai.org/ocs/index.php/WS/AAAIW12/paper/view/5203/5564
Yoo SB, Han M (2020) Temporal matching prior network for vehicle license plate detection and recognition in videos. ETRI J 42(3):411–419. https://doi.org/10.4218/etrij.2019-0245
Yun K, Kwon Y, Oh S, Moon J, Park J (2019) Vision-based garbage dumping action detection for real-world surveillance platform. ETRI J 41 (4):494–505. https://doi.org/10.4218/etrij.2018-0520
Zeng M, Nguyen LT, Yu B, Mengshoel OJ, Zhu J, Wu P, Zhang J (2014) Convolutional neural networks for human activity recognition using mobile sensors. In: 6th International conference on mobile computing applications and services. https://doi.org/10.4108/icst.mobicase.2014.257786, pp 197–205
Zhang S, He F (2020) Drcdn: learning deep residual convolutional dehazing networks. Vis Comput 36 (9):1797–1808. https://doi.org/10.1007/s00371-019-01774-8
Zhang S, He F, Ren W (2020) Nldn: Non-local dehazing network for dense haze removal. Neurocomputing 410:363–373. https://doi.org/10.1016/j.neucom.2020.06.041. http://www.sciencedirect.com/science/article/pii/S0925231220310122
Acknowledgements
This work was partly supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [1711103127, Development of Human Enhancement Technology for auditory and muscle support] and Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government (MSIT) [20ZS1100, Core Technology Research for Self-Improving Integrated Artificial Intelligence System].
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interests
The authors declare that they have no conflict of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Jeong, C.Y., Shin, H.C. & Kim, M. Sensor-data augmentation for human activity recognition with time-warping and data masking. Multimed Tools Appl 80, 20991–21009 (2021). https://doi.org/10.1007/s11042-021-10600-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-021-10600-0