Skip to main content

Toward Data Augmentation and Interpretation in Sensor-Based Fine-Grained Hand Activity Recognition

  • Conference paper
  • First Online:
Deep Learning for Human Activity Recognition (DL-HAR 2021)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1370))

Included in the following conference series:


Recognizing fine-grained hand activities has widely attracted the research community’s attention in recent years. However, rather than enriched sen-sor-based datasets of whole-body activities, there are limited data available for acceler-ator-based fine-grained hand activities. In this paper, we propose a purely convolution-based Generative Adversarial Networks (GAN) approach for data augmentation on accelerator-based temporal data of fine-grained hand activities. The approach consists of 2D-Convolution discriminator and 2D-Transposed-Convolution generator that are shown capable of learning the distribution of re-shaped sensor-based data and generating synthetic instances that well reserve the cross-axis co-relation. We evaluate the usability of synthetic data by expanding existing datasets and improving the state-of-the-art classifier’s test accuracy. The in-nature unreadable sensor-based data is interpreted by introducing visualization methods including axis-wise heatmap and model-oriented decision explanation. The experiments show that our approach can effectively improve the classifier’s test accuracy by GAN-based data augmentation while well preserving the authenticity of synthetic data.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others


  1. Alzantot, M., Chakraborty, S., Srivastava, M.: SenseGen: a deep learning architecture for synthetic sensor data generation. In: 2017 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), March 2017.

  2. Bergmann, U., Jetchev, N., Vollgraf, R.: Learning texture manifolds with the periodic spatial GAN (2017)

    Google Scholar 

  3. Berthelot, D., Schumm, T., Metz, L.: BEGAN: boundary equilibrium generative adversarial networks. arXiv abs/1703.10717 (2017)

    Google Scholar 

  4. Biswas, D., et al.: Cornet: deep learning framework for PPG-based heart rate estimation and biometric identification in ambulant environment. IEEE Trans. Biomed. Circuits Syst. 13(2), 282–291 (2019)

    Article  Google Scholar 

  5. Bulling, A., Blanke, U., Schiele, B.: A tutorial on human activity recognition using body-worn inertial sensors. ACM Comput. Surv. 46(3) (2014).

  6. Chen, K., Yao, L., Zhang, D., Wang, X., Chang, X., Nie, F.: A semisupervised recurrent convolutional attention model for human activity recognition. IEEE Trans. Neural Netw. Learn. Syst. PP, 1–10 (2019).

  7. Chen, K., Zhang, D., Yao, L., Guo, B., Yu, Z., Liu, Y.: Deep learning for sensor-based human activity recognition: overview, challenges and opportunities. arXiv preprint arXiv:2001.07416 (2020)

  8. Denton, E.L., Chintala, S., Szlam, A., Fergus, R.: Deep generative image models using a Laplacian pyramid of adversarial networks. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems 28, pp. 1486–1494. Curran Associates, Inc. (2015)

    Google Scholar 

  9. Doan, H.G., Vu, H., Tran, T.H.: Recognition of hand gestures from cyclic hand movements using spatial-temporal features. In: Proceedings of the Sixth International Symposium on Information and Communication Technology, SoICT 2015, pp. 260–267. Association for Computing Machinery, New York (2015).

  10. Gammulle, H., Denman, S., Sridharan, S., Fookes, C.: Multi-level sequence GAN for group activity recognition. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11361, pp. 331–346. Springer, Cham (2019).

    Chapter  Google Scholar 

  11. Goodfellow, I., et al.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 27, pp. 2672–2680. Curran Associates, Inc. (2014).

  12. Hammerla, N., Fisher, J., Andras, P., Rochester, L., Walker, R., Plötz, T.: PD disease state assessment in naturalistic environments using deep learning. In: Twenty-ninth AAAI Conference on Artificial Intelligence (AAAI-2015). Newcastle University (2015)

    Google Scholar 

  13. Huynh, T., Schiele, B.: Analyzing features for activity recognition. In: Proceedings of the 2005 Joint Conference on Smart Objects and Ambient Intelligence: Innovative Context-Aware Services: Usages and Technologies, sOc-EUSAI 2005, pp. 159–163. Association for Computing Machinery, New York (2005).

  14. Khan, S.S., Taati, B.: Detecting unseen falls from wearable devices using channel-wise ensemble of autoencoders. Expert Syst. Appl. 87, 280–290 (2017).

    Article  Google Scholar 

  15. Kiasari, M.A., Moirangthem, D.S., Lee, M.: Human action generation with generative adversarial networks. arXiv abs/1805.10416 (2018)

    Google Scholar 

  16. Lane, N.D., Georgiev, P.: Can deep learning revolutionize mobile sensing? In: Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications, HotMobile 2015, pp. 117–122. Association for Computing Machinery, New York (2015).

  17. Laput, G., Harrison, C.: Sensing fine-grained hand activity with smartwatches. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019 pp. 1–13. Association for Computing Machinery, New York (2019).

  18. Luna-Perejon, F., et al.: An automated fall detection system using recurrent neural networks. In: Riaño, D., Wilk, S., ten Teije, A. (eds.) AIME 2019. LNCS (LNAI), vol. 11526, pp. 36–41. Springer, Cham (2019).

    Chapter  Google Scholar 

  19. Moshiri, P., Navidan, H., Shahbazian, R., Ghorashi, S.A., Windridge, D.: Using GAN to enhance the accuracy of indoor human activity recognition. arXiv abs/2004.11228 (2020)

    Google Scholar 

  20. Münzner, S., Schmidt, P., Reiss, A., Hanselmann, M., Stiefelhagen, R., Dürichen, R.: CNN-based sensor fusion techniques for multimodal human activity recognition. In: Proceedings of the 2017 ACM International Symposium on Wearable Computers, ISWC 2017, pp. 158–165. Association for Computing Machinery, New York (2017).

  21. Ogata, M., Imai, M.: Skinwatch: skin gesture interaction for smart watch. In: Proceedings of the 6th Augmented Human International Conference, AH 2015, pp. 21–24. Association for Computing Machinery, New York (2015).

  22. Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks (2015)

    Google Scholar 

  23. Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 13–17 August 2016, pp. 1135–1144 (2016)

    Google Scholar 

  24. Shipman, F.M., Gutierrez-Osuna, R., Monteiro, C.D.D.: Identifying sign language videos in video sharing sites. ACM Trans. Access. Comput. 5(4) (2014).

  25. Tu, Y., Lin, Y., Wang, J., Kim, J.U.: Semi-supervised learning with generative adversarial networks on digital signal modulation classification. Comput. Mater. Continua 55(2), 243–254 (2018)

    Google Scholar 

  26. Wang, J., Chen, Y., Hao, S., Peng, X., Hu, L.: Deep learning for sensor-based activity recognition: a survey. Pattern Recogn. Lett. 119, 3–11 (2019)

    Article  Google Scholar 

  27. Wang, J., Chen, Y., Gu, Y., Xiao, Y., Pan, H.: SensoryGANs: an effective generative adversarial framework for sensor-based human activity recognition. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2018)

    Google Scholar 

  28. Yang, J.B., Nguyen, M.N., San, P.P., Li, X.L., Krishnaswamy, S.: Deep convolutional neural networks on multichannel time series for human activity recognition. In: Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI 2015, pp. 3995–4001. AAAI Press (2015)

    Google Scholar 

  29. Yao, S., et al.: SenseGan: enabling deep learning for internet of things with a semi-supervised framework. Proc. ACM Interact. Mob. Wearable Ubiquit. Technol. 2(3) (2018).

  30. Zhang, X., Zhu, X., Zhang, X., Zhang, N., Li, P., Wang, L.: SegGAN: semantic segmentation with generative adversarial network. In: 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM), pp. 1–5 (2018)

    Google Scholar 

  31. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2242–2251 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Jinqi Luo .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Luo, J., Li, X., Younes, R. (2021). Toward Data Augmentation and Interpretation in Sensor-Based Fine-Grained Hand Activity Recognition. In: Li, X., Wu, M., Chen, Z., Zhang, L. (eds) Deep Learning for Human Activity Recognition. DL-HAR 2021. Communications in Computer and Information Science, vol 1370. Springer, Singapore.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-16-0574-1

  • Online ISBN: 978-981-16-0575-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics