Skip to main content
SpringerLink
Log in

Daily unbalanced action recognition based on active learning

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The identification of daily activities is mainly limited by the imbalanced number of actions and the diversity of categories, which can lead to unsatisfactory classification results and difficult annotation. In this paper, a novel model is proposed to efficiently solve the mentioned problems. The model fully considers the characteristics of sample imbalance and uses the combination of Synthetic Minority Oversampling Technique and K-means clustering undersampling to achieve data balancing. After that, Active Learning is applied to query the most representative samples for labeling to reduce data annotation. Experiments show that the proposed method can balance the number of samples without increasing the computational cost of the system. Compared with the original data, the proposed model can learn the features of each class more fully after the samples are balanced, increasing the F1-score by 23%, and it can reduce the sample annotations by nearly 30% without reducing the recognition results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

The datasets generated during the current study are available in the [ECML SPHERE Challenge] repository, [https://doi.org/10.5523/bris.8gccwpx47rav19vk8x4xapcog].

References

  1. Adaimi R, Thomaz E (2019) Leveraging active learning and conditional mutual information to minimize data annotation in human activity recognition [J]. Proc ACM Interact Mob Wearable Ubiquitous Technol 3(3):1–23. https://doi.org/10.1145/3351228

    Article  Google Scholar 

  2. Akbari A, Castilla RS, Jafari R et al (2020) Using intelligent personal annotations to improve human activity recognition for movements in natural environments [J]. IEEE J Biomed Health Inform 24(9):2639–2650. https://doi.org/10.1109/JBHI.2020.2966151

    Article  Google Scholar 

  3. Alani AA, Cosma G, Taherkhani A (2020) Classifying imbalanced multi-modal sensor data for human activity recognition in a smart home using deep learning [C]. In: 2020 international joint conference on neural networks (IJCNN). IEEE, pp 1–8. https://doi.org/10.1109/IJCNN48605.2020.9207697

    Chapter  Google Scholar 

  4. Arafat MY, Hoque S, Farid DM (2017) Cluster-based under-sampling with random forest for multi-class imbalanced classification [C]. In: 2017 11th international conference on software, knowledge, information management and applications (SKIMA). IEEE, pp 1–6. https://doi.org/10.1109/SKIMA.2017.8294105

    Chapter  Google Scholar 

  5. Arafat MY, Hoque S, Xu S et al (2019) An under-sampling method with support vectors in multi-class imbalanced data classification [C]. In: 2019 13th international conference on software, knowledge, information management and applications (SKIMA). IEEE, pp 1–6. https://doi.org/10.1109/SKIMA47702.2019.8982391

    Chapter  Google Scholar 

  6. Bengar JZ, van de Weijer J, Twardowski B et al (2021) Reducing label effort: Self-supervised meets active learning [C]. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 1631–1639. https://doi.org/10.1109/ICCVW54120.2021.00188

    Chapter  Google Scholar 

  7. Bengar JZ, van de Weijer J, Fuentes LL et al (2022) Class-balanced active learning for image classification [C]. Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 1536–1545. https://doi.org/10.48550/arXiv.2110.04543

  8. Bi H, Perello-Nieto M, Santos-Rodriguez R et al (2020) Human activity recognition based on dynamic active learning [J]. IEEE J Biomed Health Inform 25(4):922–934. https://doi.org/10.1109/JBHI.2020.3013403

    Article  Google Scholar 

  9. Buda M, Maki A, Mazurowski MA (2018) A systematic study of the class imbalance problem in convolutional neural networks [J]. Neural Netw 106:249–259. https://doi.org/10.1016/j.neunet.2018.07.011

    Article  Google Scholar 

  10. Bulling A, Blanke U, Schiele B (2014) A tutorial on human activity recognition using body-worn inertial sensors [J]. ACM Comput Surv (CSUR) 46(3):1–33. https://doi.org/10.1145/2499621

    Article  Google Scholar 

  11. Chawla NV, Bowyer KW, Hall LO et al (2002) SMOTE: synthetic minority over-sampling technique [J]. J Artif Intell Res 16:321–357. https://doi.org/10.1613/jair.953

    Article  Google Scholar 

  12. Chen Z, Jiang C, Xiang S et al (2019) Smartphone sensor-based human activity recognition using feature fusion and maximum full a posteriori [J]. IEEE Trans Instrum Meas 69(7):3992–4001. https://doi.org/10.1109/TIM.2019.2945467

    Article  Google Scholar 

  13. Chen K, Zhang D, Yao L et al (2021) Deep learning for sensor-based human activity recognition: overview, challenges, and opportunities [J]. ACM Comput Surv (CSUR) 54(4):1–40. https://doi.org/10.1145/3447744

    Article  Google Scholar 

  14. Choi J, Yi KM, Kim J et al (2021) Vab-al: Incorporating class imbalance and difficulty with variational bayes for active learning [C]. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6749–6758. https://doi.org/10.48550/arXiv.2003.11249

  15. Cipolla E, Infantino I, Maniscalco U et al (2017) Indoor actions classification through long short term memory neural networks [C]. In: International conference on image analysis and processing. Springer, Cham, pp 435–444. https://doi.org/10.1007/978-3-319-68560-1_39

    Chapter  Google Scholar 

  16. Elsts A, Twomey N, McConville R et al (2020) Energy-efficient activity recognition framework using wearable accelerometers [J]. J Netw Comput Appl 168:s. https://doi.org/10.1016/j.jnca.2020.102770

    Article  Google Scholar 

  17. Gao G, Li Z, Huan Z et al (2021) Human behavior recognition model based on feature and classifier selection [J]. Sensors 21(23):7791. https://doi.org/10.3390/s21237791

    Article  Google Scholar 

  18. Guo S, Liu Y, Chen R et al (2019) Improved SMOTE algorithm to deal with imbalanced activity classes in smart homes [J]. Neural Process Lett 50(2):1503–1526. https://doi.org/10.1007/s11063-018-9940-3

    Article  Google Scholar 

  19. Hamad RA, Kimura M, Lundström J (2020) Efficacy of imbalanced data handling methods on deep learning for smart homes environments [J]. SN Comput Sci 1:1–10. https://doi.org/10.1007/s42979-020-00211-1

  20. Ho SS, Wechsler H (2008) Query by transduction [J]. IEEE Trans Pattern Anal Mach Intell 30(9):1557–1571. https://doi.org/10.1109/TPAMI.2007.70811

    Article  Google Scholar 

  21. Huan Z, Wei C, Li GH (2018) Outlier detection in wireless sensor networks using model selection-based support vector data descriptions [J]. Sensors 18(12):4328. https://doi.org/10.3390/s18124328

    Article  Google Scholar 

  22. Huan Z, Chen X, Lv S et al (2019) Gait recognition of acceleration sensor for smart phone based on multiple classifier fusion [J]. Math Probl Eng 2019. https://doi.org/10.1155/2019/6471532

  23. Huan Z, Lv S, Hou Z et al (2020) An evaluation strategy for the symmetry and consistency of lower limb segments during upper limb loading [J]. IEEE Sensors J 21(5):6440–6449. https://doi.org/10.1109/JSEN.2020.3039865

    Article  Google Scholar 

  24. Ijaz MF, Alfian G, Syafrudin M et al (2018) Hybrid prediction model for type 2 diabetes and hypertension using DBSCAN-based outlier detection, synthetic minority over sampling technique (SMOTE), and random forest [J]. Appl Sci 8(8):1325. https://doi.org/10.3390/app8081325

    Article  Google Scholar 

  25. Ijaz MF, Attique M, Son Y (2020) Data-driven cervical cancer prediction model with outlier detection and over-sampling methods [J]. Sensors 20(10):2809. https://doi.org/10.3390/s20102809

    Article  Google Scholar 

  26. Janicka M, Lango M, Stefanowski J (2019) Using information on class interrelations to improve classification of multiclass imbalanced data: a new resampling algorithm [J]. Int J Appl Math Comput Sci 29(4). https://doi.org/10.2478/amcs-2019-0057

  27. Koziarski M (2020) Radial-based undersampling for imbalanced data classification [J]. Pattern Recogn 102:107262. https://doi.org/10.1016/j.patcog.2020.107262

    Article  Google Scholar 

  28. Koziarski M, Woźniak M, Krawczyk B (2020) Combined cleaning and resampling algorithm for multi-class imbalanced data with label noise [J]. Knowl-Based Syst 204:106223. https://doi.org/10.1016/j.knosys.2020.106223

    Article  Google Scholar 

  29. Kumar Y, Koul A, Singla R et al (2022) Artificial intelligence in disease diagnosis: a systematic literature review, synthesizing framework and future research agenda [J]. J Ambient Intell Humaniz Comput:1–28. https://doi.org/10.1007/s12652-021-03612-z

  30. Kwon H, Abowd GD, Plötz T (2019) Handling annotation uncertainty in human activity recognition [C]. In: Proceedings of the 23rd international symposium on wearable computers, pp 109–117. https://doi.org/10.1145/3341163.3347744

    Chapter  Google Scholar 

  31. Lewis DD, Gale WA (1994) A sequential algorithm for training text classifiers [C]//SIGIR’94. Springer, London, pp 3–12. https://doi.org/10.1145/219587.219592

    Book  Google Scholar 

  32. Liu M, Dong M, Jing C (2021) A modified real-value negative selection detector-based oversampling approach for multiclass imbalance problems [J]. Inf Sci 556:160–176. https://doi.org/10.1016/j.ins.2020.12.058

    Article  MathSciNet  Google Scholar 

  33. Malki Z, Atlam E, Dagnew G et al (2020) Bidirectional residual LSTM-based human activity recognition [J]. Comput Inf Sci 13(3):40. https://doi.org/10.5539/cis.v13n3p40

    Article  Google Scholar 

  34. Narasimman G, Lu K, Raja A et al A* HAR: a new benchmark towards semi-supervised learning for class-imbalanced human activity recognition [J]. arXiv preprint arXiv:2101.04859, 2021. https://doi.org/10.48550/arXiv.2101.04859

  35. Nekooeimehr I, Lai-Yuen SK (2016) Adaptive semi-unsupervised weighted oversampling (A-SUWO) for imbalanced datasets [J]. Expert Syst Appl 46:405–416. https://doi.org/10.1016/j.eswa.2015.10.031

    Article  Google Scholar 

  36. Oh S, Ashiquzzaman A, Lee D et al (2021) Study on human activity recognition using semi-supervised active transfer learning [J]. Sensors 21(8):2760. https://doi.org/10.3390/s21082760

    Article  Google Scholar 

  37. Pruengkarn R, Wong KW, Fung CC  (2017) Multiclass imbalanced classification using fuzzy C-mean and SMOTE with fuzzy support vector machine [C] International conference on neural information processing. Springer, Cham: 67–75. https://doi.org/10.1007/978-3-319-70139-4_7

  38. Ramanujam E, Perumal T, Padmavathi S (2021) Human activity recognition with smartphone and wearable sensors using deep learning techniques: a review [J]. IEEE Sensors J. https://doi.org/10.1109/JSEN.2021.3069927

  39. Song J, Huang X, Qin S et al (2016) A bi-directional sampling based on K-means method for imbalance text classification [C]. In: 2016 IEEE/ACIS 15th international conference on computer and information science (ICIS). IEEE, pp 1–5. https://doi.org/10.1109/ICIS.2016.7550920

    Chapter  Google Scholar 

  40. Srinivasu PN, SivaSai JG, Ijaz MF et al (2021) Classification of skin disease using deep learning neural networks with MobileNet V2 and LSTM [J]. Sensors 21(8):2852. https://doi.org/10.3390/s21082852

    Article  Google Scholar 

  41. Taherkhani A, Cosma G, Alani AA et al (2018) Activity recognition from multi-modal sensor data using a deep convolutional neural network [C]. In: Science and information conference. Springer, Cham, pp 203–218. https://doi.org/10.1007/978-3-030-01177-2_15

    Chapter  Google Scholar 

  42. Twomey N, Diethe T, Kull M et al (2016) The SPHERE challenge: Activity recognition with multimodal sensor data [J]. arXiv preprint arXiv:1603.00797. https://doi.org/10.48550/arXiv.1603.00797

  43. Woznowski P, Burrows A, Diethe T et al (2017) SPHERE: a sensor platform for healthcare in a residential environment [M]. In: Designing, developing, and facilitating smart cities. Springer, Cham, pp 315–333. https://doi.org/10.1007/978-3-319-44924-1_14

    Chapter  Google Scholar 

  44. Wu D, Wang Z, Chen Y et al (2016) Mixed-kernel based weighted extreme learning machine for inertial sensor based human activity recognition with imbalanced dataset [J]. Neurocomputing 190:35–49. https://doi.org/10.1016/j.neucom.2015.11.095

    Article  Google Scholar 

  45. Xie W, Liang G, Dong Z et al (2019) An improved oversampling algorithm based on the samples’ selection strategy for classifying imbalanced data [J]. Math Probl Eng 2019. https://doi.org/10.1155/2019/3526539

  46. Zhao J, Jin J, Chen S et al (2020) A weighted hybrid ensemble method for classifying imbalanced data [J]. Knowl-Based Syst 203:106087. https://doi.org/10.1016/j.knosys.2020.106087

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant 61772248. The project name is Sensing Theory and Key Technologies of Big Behavioral Data for Wearable Computing. (Corresponding author: Zhan Huan).

Author information

Authors and Affiliations

  1. School of Computer and Artificial Intelligence, School of Big Data, Changzhou University, Changzhou, 213164, China

    Yan Liu & Bangwen Zhou

  2. School of Microelectronics and Control Engineering, Changzhou University, Changzhou, 213164, China

    Zhixin Li, Zhan Huan, Shiyi Shen & Shan Gao

Authors
  1. Yan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhixin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhan Huan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bangwen Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shiyi Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shan Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhan Huan.

Ethics declarations

Conflict of interest

The authors declare that there are no competing interests regarding the publication of this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Y., Li, Z., Huan, Z. et al. Daily unbalanced action recognition based on active learning. Multimed Tools Appl 83, 16255–16274 (2024). https://doi.org/10.1007/s11042-023-16181-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16181-4

Keywords

Search

Navigation