Abstract
Wearable sensor based human activity recognition is a challenging problem due to difficulty in modeling spatial and temporal dependencies of sensor signals. Recognition models in closed-set assumption are forced to yield members of known activity classes as prediction. However, activity recognition models can encounter an unseen activity due to body-worn sensor malfunction or disability of the subject performing the activities. This problem can be addressed through modeling solution according to the assumption of open-set recognition. Hence, the proposed self attention based approach combines data hierarchically from different sensor placements across time to classify closed-set activities and it obtains notable performance improvement over state-of-the-art models on five publicly available datasets. The decoder in this autoencoder architecture incorporates self-attention based feature representations from encoder to detect unseen activity classes in open-set recognition setting. Furthermore, attention maps generated by the hierarchical model demonstrate explainable selection of features in activity recognition. We conduct extensive leave one subject out validation experiments that indicate significantly improved robustness to noise and subject specific variability in body-worn sensor signals. The source code is available at: github.com/saif-mahmud/hierarchical-attention-HAR.
M. Tanjid Hasan Tonmoy and Saif Mahmud—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Al Machot, F., Elkobaisi, M.R., Kyamakya, K.: Zero-shot human activity recognition using non-visual sensors. Sensors 20(3), 825 (2020)
An, J., Cho, S.: Variational autoencoder based anomaly detection using reconstruction probability. Special Lecture on IE, vol. 2, no. 1, pp. 1–18 (2015)
Bachlin, M., et al.: Wearable assistant for Parkinson’s disease patients with the freezing of gait symptom. IEEE Trans. Inf Technol. Biomed. 14(2), 436–443 (2010)
Cheng, W., Erfani, S., Zhang, R., Kotagiri, R.: Predicting complex activities from ongoing multivariate time series. In: IJCAI 2018, pp. 3322–3328 (2018)
Du, L., Tan, Y.: A novel human activity recognition and prediction in smart home based on interaction. Sensors 19(20), 4474 (2019)
Gao, S., et al.: Classifying cancer pathology reports with hierarchical self-attention networks. Artif. Intell. Med. 101, 101726 (2019)
Guan, Y., Plötz, T.: Ensembles of deep lstm learners for activity recognition using wearables. ACM IMWUT (2017)
Hammerla, N.Y., Halloran, S., Plötz, T.: Deep, convolutional, and recurrent models for human activity recognition using wearables. arXiv preprint arXiv:1604.08880 (2016)
Haresamudram, H., Anderson, D.V., Plötz, T.: On the role of features in human activity recognition. In: Proceedings of the 23rd International Symposium on Wearable Computers, ISWC 2019. ACM (2019)
He, Y., Wang, J., Niu, W., Caverlee, J.: A hierarchical self-attentive model for recommending user-generated item lists. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 1481–1490 (2019)
Kwon, H., Abowd, G.D., Plötz, T.: Adding structural characteristics to distribution-based accelerometer representations for activity recognition using wearables. In: Proceedings of the 2018 ACM ISWC. ACM (2018)
Ma, H., Li, W., Zhang, X., Gao, S., Lu, S.: AttnSense: multi-level attention mechanism for multimodal human activity recognition. In: Proceedings of the IJCAI 2019, pp. 3109–3115 (2019)
Mahmud, S., et al.: Human activity recognition from wearable sensor data using self-attention (ECAI 2020) (2020)
Meng, L., et al.: Interpretable spatio-temporal attention for video action recognition. In: Proceedings of the IEEE International Conference on Computer Vision Workshops (2019)
Murahari, V.S., Plötz, T.: On attention models for human activity recognition. In: 2018 ACM International Symposium on Wearable Computers, pp. 100–103. ACM (2018)
OrdÃñez, F., Roggen, D.: Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors 16, 115 (2016)
Oza, P., Patel, V.M.: C2ae: class conditioned auto-encoder for open-set recognition. In: Proceedings of the IEEE CVPR, pp. 2307–2316 (2019)
Peng, L., Chen, L., Ye, Z., Zhang, Y.: Aroma: a deep multi-task learning based simple and complex human activity recognition method using wearable sensors. In: Proceedings of ACM on Interactive, Mobile, Wearable Ubiquitous Technologies (2018)
Plötz, T., Hammerla, N.Y., Olivier, P.L.: Feature learning for activity recognition in ubiquitous computing. In: Twenty-Second International Joint Conference on Artificial Intelligence (2011)
Qian, H., Pan, S.J., Da, B., Miao, C.: A novel distribution-embedded neural network for sensor-based activity recognition. In: Proceedings of the IJCAI 2019 (2019)
Ramakrishnan, K., et al.: Identifying interpretable action concepts in deep networks. In: CVPR Workshops, pp. 12–15 (2019)
Reiss, A., Stricker, D.: Introducing a new benchmarked dataset for activity monitoring. In: Proceedings of the 16th Annual International Symposium on Wearable Computers (ISWC) (2012)
Roggen, D., et al.: Collecting complex activity datasets in highly rich networked sensor environments. In: Seventh International Conference on Networked Sensing Systems (INSS) (2010)
Stiefmeier, T., Roggen, D., Troster, G., Ogris, G., Lukowicz, P.: Wearable activity tracking in car manufacturing. IEEE Pervas. Comput. 7, 42–50 (2008)
Tian, G., Yin, J., Han, X., Yu, J.: A novel human activity recognition method using joint points information. Jiqiren/Robot 36, 285–292 (2014)
Vasilev, A., et al.: q-space novelty detection with variational autoencoders. In: Bonet-Carne, E., Hutter, J., Palombo, M., Pizzolato, M., Sepehrband, F., Zhang, F. (eds.) Computational Diffusion MRI. MV, pp. 113–124. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-52893-5_10
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Yao, S., et a.: Sadeepsense: self-attention deep learning framework for heterogeneous on-device sensors in internet of things applications. In: IEEE INFOCOM 2019-IEEE Conference on Computer Communications, pp. 1243–1251. IEEE (2019)
Zeng, M., et al.: Understanding and improving recurrent networks for human activity recognition by continuous attention. In: ACM ISWC 2018. ACM (2018)
Zhang, M., Sawchuk, A.: USC-had: a daily activity dataset for ubiquitous activity recognition using wearable sensors. In: 2012 ACM Conference on Ubiquitous Computing, pp. 1036–1043 (2012)
Zheng, Z., Shi, L., Wang, C., Sun, L., Pan, G.: LSTM with uniqueness attention for human activity recognition. In: Tetko, I.V., Kůrková, V., Karpov, P., Theis, F. (eds.) ICANN 2019. LNCS, vol. 11729, pp. 498–509. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30508-6_40
Acknowledgment
This project is supported by ICT Division, Government of Bangladesh, and Independent University, Bangladesh (IUB).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Tonmoy, M.T.H., Mahmud, S., Mahbubur Rahman, A.K.M., Ashraful Amin, M., Ali, A.A. (2021). Hierarchical Self Attention Based Autoencoder for Open-Set Human Activity Recognition. In: Karlapalem, K., et al. Advances in Knowledge Discovery and Data Mining. PAKDD 2021. Lecture Notes in Computer Science(), vol 12714. Springer, Cham. https://doi.org/10.1007/978-3-030-75768-7_28
Download citation
DOI: https://doi.org/10.1007/978-3-030-75768-7_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-75767-0
Online ISBN: 978-3-030-75768-7
eBook Packages: Computer ScienceComputer Science (R0)