Abstract
Recognizing fine-grained hand activities has widely attracted the research community’s attention in recent years. However, rather than enriched sen-sor-based datasets of whole-body activities, there are limited data available for acceler-ator-based fine-grained hand activities. In this paper, we propose a purely convolution-based Generative Adversarial Networks (GAN) approach for data augmentation on accelerator-based temporal data of fine-grained hand activities. The approach consists of 2D-Convolution discriminator and 2D-Transposed-Convolution generator that are shown capable of learning the distribution of re-shaped sensor-based data and generating synthetic instances that well reserve the cross-axis co-relation. We evaluate the usability of synthetic data by expanding existing datasets and improving the state-of-the-art classifier’s test accuracy. The in-nature unreadable sensor-based data is interpreted by introducing visualization methods including axis-wise heatmap and model-oriented decision explanation. The experiments show that our approach can effectively improve the classifier’s test accuracy by GAN-based data augmentation while well preserving the authenticity of synthetic data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Alzantot, M., Chakraborty, S., Srivastava, M.: SenseGen: a deep learning architecture for synthetic sensor data generation. In: 2017 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), March 2017. https://doi.org/10.1109/PERCOMW.2017.7917555
Bergmann, U., Jetchev, N., Vollgraf, R.: Learning texture manifolds with the periodic spatial GAN (2017)
Berthelot, D., Schumm, T., Metz, L.: BEGAN: boundary equilibrium generative adversarial networks. arXiv abs/1703.10717 (2017)
Biswas, D., et al.: Cornet: deep learning framework for PPG-based heart rate estimation and biometric identification in ambulant environment. IEEE Trans. Biomed. Circuits Syst. 13(2), 282–291 (2019)
Bulling, A., Blanke, U., Schiele, B.: A tutorial on human activity recognition using body-worn inertial sensors. ACM Comput. Surv. 46(3) (2014). https://doi.org/10.1145/2499621
Chen, K., Yao, L., Zhang, D., Wang, X., Chang, X., Nie, F.: A semisupervised recurrent convolutional attention model for human activity recognition. IEEE Trans. Neural Netw. Learn. Syst. PP, 1–10 (2019). https://doi.org/10.1109/TNNLS.2019.2927224
Chen, K., Zhang, D., Yao, L., Guo, B., Yu, Z., Liu, Y.: Deep learning for sensor-based human activity recognition: overview, challenges and opportunities. arXiv preprint arXiv:2001.07416 (2020)
Denton, E.L., Chintala, S., Szlam, A., Fergus, R.: Deep generative image models using a Laplacian pyramid of adversarial networks. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems 28, pp. 1486–1494. Curran Associates, Inc. (2015)
Doan, H.G., Vu, H., Tran, T.H.: Recognition of hand gestures from cyclic hand movements using spatial-temporal features. In: Proceedings of the Sixth International Symposium on Information and Communication Technology, SoICT 2015, pp. 260–267. Association for Computing Machinery, New York (2015). https://doi.org/10.1145/2833258.2833301
Gammulle, H., Denman, S., Sridharan, S., Fookes, C.: Multi-level sequence GAN for group activity recognition. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11361, pp. 331–346. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20887-5_21
Goodfellow, I., et al.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 27, pp. 2672–2680. Curran Associates, Inc. (2014). http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf
Hammerla, N., Fisher, J., Andras, P., Rochester, L., Walker, R., Plötz, T.: PD disease state assessment in naturalistic environments using deep learning. In: Twenty-ninth AAAI Conference on Artificial Intelligence (AAAI-2015). Newcastle University (2015)
Huynh, T., Schiele, B.: Analyzing features for activity recognition. In: Proceedings of the 2005 Joint Conference on Smart Objects and Ambient Intelligence: Innovative Context-Aware Services: Usages and Technologies, sOc-EUSAI 2005, pp. 159–163. Association for Computing Machinery, New York (2005). https://doi.org/10.1145/1107548.1107591
Khan, S.S., Taati, B.: Detecting unseen falls from wearable devices using channel-wise ensemble of autoencoders. Expert Syst. Appl. 87, 280–290 (2017). https://doi.org/10.1016/j.eswa.2017.06.011
Kiasari, M.A., Moirangthem, D.S., Lee, M.: Human action generation with generative adversarial networks. arXiv abs/1805.10416 (2018)
Lane, N.D., Georgiev, P.: Can deep learning revolutionize mobile sensing? In: Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications, HotMobile 2015, pp. 117–122. Association for Computing Machinery, New York (2015). https://doi.org/10.1145/2699343.2699349
Laput, G., Harrison, C.: Sensing fine-grained hand activity with smartwatches. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019 pp. 1–13. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3290605.3300568
Luna-Perejon, F., et al.: An automated fall detection system using recurrent neural networks. In: Riaño, D., Wilk, S., ten Teije, A. (eds.) AIME 2019. LNCS (LNAI), vol. 11526, pp. 36–41. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-21642-9_6
Moshiri, P., Navidan, H., Shahbazian, R., Ghorashi, S.A., Windridge, D.: Using GAN to enhance the accuracy of indoor human activity recognition. arXiv abs/2004.11228 (2020)
Münzner, S., Schmidt, P., Reiss, A., Hanselmann, M., Stiefelhagen, R., Dürichen, R.: CNN-based sensor fusion techniques for multimodal human activity recognition. In: Proceedings of the 2017 ACM International Symposium on Wearable Computers, ISWC 2017, pp. 158–165. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3123021.3123046
Ogata, M., Imai, M.: Skinwatch: skin gesture interaction for smart watch. In: Proceedings of the 6th Augmented Human International Conference, AH 2015, pp. 21–24. Association for Computing Machinery, New York (2015). https://doi.org/10.1145/2735711.2735830
Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks (2015)
Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 13–17 August 2016, pp. 1135–1144 (2016)
Shipman, F.M., Gutierrez-Osuna, R., Monteiro, C.D.D.: Identifying sign language videos in video sharing sites. ACM Trans. Access. Comput. 5(4) (2014). https://doi.org/10.1145/2579698
Tu, Y., Lin, Y., Wang, J., Kim, J.U.: Semi-supervised learning with generative adversarial networks on digital signal modulation classification. Comput. Mater. Continua 55(2), 243–254 (2018)
Wang, J., Chen, Y., Hao, S., Peng, X., Hu, L.: Deep learning for sensor-based activity recognition: a survey. Pattern Recogn. Lett. 119, 3–11 (2019)
Wang, J., Chen, Y., Gu, Y., Xiao, Y., Pan, H.: SensoryGANs: an effective generative adversarial framework for sensor-based human activity recognition. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2018)
Yang, J.B., Nguyen, M.N., San, P.P., Li, X.L., Krishnaswamy, S.: Deep convolutional neural networks on multichannel time series for human activity recognition. In: Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI 2015, pp. 3995–4001. AAAI Press (2015)
Yao, S., et al.: SenseGan: enabling deep learning for internet of things with a semi-supervised framework. Proc. ACM Interact. Mob. Wearable Ubiquit. Technol. 2(3) (2018). https://doi.org/10.1145/3264954
Zhang, X., Zhu, X., Zhang, X., Zhang, N., Li, P., Wang, L.: SegGAN: semantic segmentation with generative adversarial network. In: 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM), pp. 1–5 (2018)
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2242–2251 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Luo, J., Li, X., Younes, R. (2021). Toward Data Augmentation and Interpretation in Sensor-Based Fine-Grained Hand Activity Recognition. In: Li, X., Wu, M., Chen, Z., Zhang, L. (eds) Deep Learning for Human Activity Recognition. DL-HAR 2021. Communications in Computer and Information Science, vol 1370. Springer, Singapore. https://doi.org/10.1007/978-981-16-0575-8_3
Download citation
DOI: https://doi.org/10.1007/978-981-16-0575-8_3
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-16-0574-1
Online ISBN: 978-981-16-0575-8
eBook Packages: Computer ScienceComputer Science (R0)