Skip to main content

Advertisement

SpringerLink
Log in

Personalized human activity recognition using deep learning and edge-cloud architecture

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Human activity recognition is a thriving field with many applications in several domains. It relies on well-trained artificial intelligence models to provide accurate real-time predictions of various human movements and activities. Human activity recognition utilizes various types of sensors such as video cameras, fixed motion sensors, and those found in personal smart edge devices such as accelerometers and gyroscopes. The latter sensors capture motion as time-series data, following a specific pattern for each movement. However, movements for some users may vary from these patterns, which limit the efficacy of using a generic model. This paper proposes a human activity recognition architecture that utilizes deep learning models using time-series data. It applies incremental learning for building personalized models derived from a well-trained model. The architecture uses edge devices for model prediction and the cloud for model training. Performing the prediction on edge devices reduces the network overhead as well as the load on the cloud. We tested our approach on a publicly available dataset containing samples for daily living activities and fall states. The results show that building a personalized model from a well-trained model significantly improves the prediction accuracy. Moreover, deploying a light version of the model on edge devices maintains prediction accuracy and provides comparable response times to the original model on the cloud.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  • Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M et al (2016) Tensorflow: a system for large-scale machine learning. In: 12th USENIX symposium on operating systems design and implementation (OSDI 16), pp 265–283

  • Agarwal P, Alam M (2020) A lightweight deep learning model for human activity recognition on edge devices. Procedia Comput Sci 167:2364–2373

    Google Scholar 

  • Akbari A, Jafari R (2020) Personalizing activity recognition models through quantifying different types of uncertainty using wearable sensors. IEEE Trans Biomed Eng 67(9):2530–2541

    Google Scholar 

  • Alawneh L, Alsarhan T, Al-Zinati M, Al-Ayyoub M, Jararweh Y, Lu H (2021) Enhancing human activity recognition using deep learning and time series augmented data. J Ambient Intell Human Comput 12:10565–10580

    Google Scholar 

  • Anguita D, Ghio A, Oneto L, Parra X, Reyes-Ortiz JL (2013) A public domain dataset for human activity recognition using smartphones. In: Esann, vol 3, p 3

  • Anjana W, Nirmalie W (2020) Personalised meta-learning for human activity recognition with few-data. In: International conference on innovative techniques and applications of artificial intelligence. Springer, pp 79–93

  • Ashkan Y, Caleb F, Tam N, Krishna K, Fatemeh J, Amirreza N, Jian K, Jue JP (2019) All one needs to know about fog computing and related edge computing paradigms: a complete survey. J Syst Archit 98:289–330

    Google Scholar 

  • Bisong E (2019) Google colaboratory. In: Building machine learning and deep learning models on Google Cloud platform. Springer, pp 59–64

  • Bulling A, Blanke U, Schiele B (2014) A tutorial on human activity recognition using body-worn inertial sensors. ACM Comput Surv (CSUR) 46(3):1–33

    Google Scholar 

  • Cao Y, Chen S, Hou P, Brown D (2015) Fast: a fog computing assisted distributed analytics system to monitor fall for stroke mitigation. In: 2015 IEEE international conference on networking, architecture and storage (NAS). IEEE, pp 2–11

  • Casale P, Pujol O, Radeva P (2011) Human activity recognition from accelerometer data using a wearable device. In: Iberian conference on pattern recognition and image analysis. Springer, pp 289–296

  • Chang X, Nie F, Wang S, Yang Y, Zhou X, Zhang C (2015) Compound rank-\(k\) projections for bilinear analysis. IEEE Trans Neural Netw Learn Syst 27(7):1502–1513

    MathSciNet  Google Scholar 

  • Chen Z, Zhu Q, Soh YC, Zhang L (2017) Robust human activity recognition using smartphone sensors via ct-pca and online svm. IEEE Trans Ind Inform 13(6):3070–3080

    Google Scholar 

  • Chen K, Yao L, Zhang D, Wang X, Chang X, Nie F (2019) A semisupervised recurrent convolutional attention model for human activity recognition. IEEE Trans Neural Netw Learn Syst 31(5):1747–1756

    Google Scholar 

  • Chen Y, Zhong K, Zhang J, Sun Q, Zhao X (2016) Lstm networks for mobile human activity recognition. In: 2016 International conference on artificial intelligence: technologies and applications. Atlantis Press, pp 50–53

  • Dhiman C, Vishwakarma DK (2019) A review of state-of-the-art techniques for abnormal human activity recognition. Eng Appl Artif Intell 77:21–45

    Google Scholar 

  • Chun B-G, Ihm S, Maniatis P, Naik M, Patti A (2011) Clonecloud: elastic execution between mobile device and cloud. In: Proceedings of the sixth conference on computer systems, pp 301–314

  • Concone F, Re GL, Morana M (2019) A fog-based application for human activity recognition using personal smart devices. ACM Trans Internet Technol (TOIT) 19(2):1–20

    Google Scholar 

  • Daniel WK (2014) Challenges on privacy and reliability in cloud computing security. In: 2014 international conference on information science, electronics and electrical engineering, vol 2. IEEE, pp 1181–1187

  • Fawaz HI, Forestier G, Weber J, Idoumghar L, Muller P-A (2019) Deep learning for time series classification: a review. Data Min Knowl Discov 33(4):917–963

    MathSciNet  MATH  Google Scholar 

  • Ferrari A, Micucci D, Mobilio M, Napoletano P (2020) On the personalization of classification models for human activity recognition. IEEE Access 8:32066–32079

    Google Scholar 

  • Gers FA, Schraudolph NN, Schmidhuber J (2002) Learning precise timing with lstm recurrent networks. J Mach Learn Res 3:115–143

    MathSciNet  MATH  Google Scholar 

  • Golestani N, Moghaddam M (2020) Human activity recognition using magnetic induction-based motion signals and deep recurrent neural networks. Nat Commun 11(1):1–11

    Google Scholar 

  • Gravina R, Ma C, Pace P, Aloi G, Russo W, Li W, Fortino G (2017) Cloud-based activity-aaservice cyber-physical framework for human activity monitoring in mobility. Future Gener Comput Syst 75:158–171

    Google Scholar 

  • Han J, Bhanu B (2005) Human activity recognition in thermal infrared imagery. In: 2005 IEEE Computer Society conference on computer vision and pattern recognition (CVPR’05)-workshops. IEEE, p 17

  • Hassan MM, Uddin MZ, Mohamed A, Almogren A (2018) A robust human activity recognition system using smartphone sensors and deep learning. Future Gener Comput Syst 81:307–313

    Google Scholar 

  • Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Google Scholar 

  • Hüsken M, Stagge P (2003) Recurrent neural networks for time series classification. Neurocomputing 50:223–235

    MATH  Google Scholar 

  • Jayaraman PP, Gomes JB, Nguyen HL, Abdallah ZS, Krishnaswamy S, Zaslavsky A (2014) Cardap: a scalable energy-efficient context aware distributed mobile data analytics platform for the fog. In: East European conference on advances in databases and information systems. Springer, pp 192–206

  • Jha S, Schiemer M, Zambonelli F, Ye J (2021) Continual learning in sensor-based human activity recognition: an empirical benchmark analysis. Inf Sci 575:1–21

    MathSciNet  Google Scholar 

  • Ketkar N (2017) Introduction to keras. In: Deep learning with Python. Springer, pp 97–111

  • Kim E, Helal S, Cook D (2009) Human activity recognition and pattern discovery. IEEE Pervasive Comput 9(1):48–53

    Google Scholar 

  • Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980

  • Lehrig S, Eikerling H, Becker S (2015) Scalability, elasticity, and efficiency in cloud computing: a systematic literature review of definitions and metrics. In: Proceedings of the 11th international ACM SIGSOFT conference on quality of software architectures, pp 83–92

  • Li Z, Nie F, Chang X, Nie L, Zhang H, Yang Y (2018) Rank-constrained spectral clustering with flexible embedding. IEEE Trans Neural Netw Learn Syst 29(12):6073–6082

    MathSciNet  Google Scholar 

  • Li Z, Nie F, Chang X, Yang Y, Zhang C, Sebe N (2018) Dynamic affinity graph construction for spectral clustering using multiple features. IEEE Trans Neural Netw Learn Syst 29(12):6323–6332

    MathSciNet  Google Scholar 

  • Li X, He Y, Jing X (2019) A survey of deep learning-based human activity recognition in radar. Remote Sens 11(9):1068

    Google Scholar 

  • Li Z, Yao L, Chang X, Zhan K, Sun J, Zhang H (2019) Zero-shot event detection via event-adaptive concept relevance mining. Pattern Recogn 88:595–603

    Google Scholar 

  • Liu C-L, Hsaio W-H, Yao-Chung T (2018) Time series classification with multivariate convolutional neural network. IEEE Trans Ind Electron 66(6):4788–4797

    Google Scholar 

  • Lockhart JW, Pulickal T, Weiss GM (2012) Applications of mobile activity recognition. In: Proceedings of the 2012 ACM conference on ubiquitous computing, pp 1054–1058

  • Maheshwari S, Raychaudhuri D, Seskar I, Bronzino F (2018) Scalability and performance evaluation of edge cloud systems for latency constrained applications. In: 2018 IEEE/ACM symposium on edge computing (SEC). IEEE, pp 286–299

  • Mazankiewicz A, Böhm K, Bergés M (2020) Incremental real-time personalization in human activity recognition using domain adaptive batch normalization. Proc ACM Interact Mobile Wearable Ubiquitous Technol 4(4):1–20

    Google Scholar 

  • Medsker LR, Jain LC (2001) Recurrent neural networks. Des Appl 5:64–67

    Google Scholar 

  • Mehdiyev N, Lahann J, Emrich A, Enke D, Fettke P, Loos P (2017) Time series classification using deep learning for process planning: a case from the process industry. Procedia Comput Sci 114:242–249

    Google Scholar 

  • Micucci D, Mobilio M, Napoletano P (2017) Unimib shar: a dataset for human activity recognition using acceleration data from smartphones. Appl Sci 7(10):1101

    Google Scholar 

  • Nwankpa CE, Ijomah W, Gachagan A, Marshall S (2021) Activation functions: comparison of trends in practice and research for deep learning. In: 2nd International Conference on Computational Sciences and Technology, pp 124–133

  • Ogbuabor G, La R (2018) Human activity recognition for healthcare using smartphones. In: Proceedings of the 2018 10th international conference on machine learning and computing, pp 41–46

  • Plötz T, Hammerla NY, Olivier PL (2011) Feature learning for activity recognition in ubiquitous computing. In: Twenty-second international joint conference on artificial intelligence

  • Polikar R, Upda L, Upda SS, Honavar V (2001) Learn++: an incremental learning algorithm for supervised neural networks. IEEE Trans Syst Man Cybern Part C (Appl Rev) 31(4):497–508

    Google Scholar 

  • Ravi D, Wong C, Lo B, Yang G-Z (2016) A deep learning approach to on-node sensor data analytics for mobile or wearable devices. IEEE J Biomed Health Inform 21(1):56–64

    Google Scholar 

  • Ren J, Zhang D, He S, Zhang Y, Li T (2019) A survey on end-edge-cloud orchestrated network computing paradigms: transparent computing, mobile edge computing, fog computing, and cloudlet. ACM Comput Surv (CSUR) 52(6):1–36

    Google Scholar 

  • Ren P, Xiao Y, Chang X, Huang P-Y, Li Z, Chen X, Wang X (2021) A comprehensive survey of neural architecture search: challenges and solutions. ACM Comput Surv (CSUR) 54(4):1–34

    Google Scholar 

  • Robertson N, Reid I (2006) A general method for human activity recognition in video. Comput Vis Image Underst 104(2–3):232–248

    Google Scholar 

  • Rokni SA, Nourollahi M, Ghasemzadeh H (2018) Personalized human activity recognition using convolutional neural networks. In: Proceedings of the AAAI conference on artificial intelligence, vol 32

  • Ronao CA, Cho S-B (2016) Human activity recognition with smartphone sensors using deep learning neural networks. Expert Syst Appl 59:235–244

    Google Scholar 

  • Rubén S-S, Jaime L-T, Beatriz M-G, Pardo JM (2016) Segmenting human activities based on hmms using smartphone inertial sensors. Pervasive Mobile Comput 30:84–96

    Google Scholar 

  • Sadiq S, Nirmalie W, Stewart M, and Kay C (2017) KNN sampling for personalised human activity recognition. In: International conference on case-based reasoning. Springer, pp 330–344

  • Satyanarayanan M (2017) The emergence of edge computing. Computer 50(1):30–39

    Google Scholar 

  • Shanhe Y, Cheng L, Qun L (2015) A survey of fog computing: concepts, applications and issues. In: Proceedings of the 2015 workshop on mobile big data, pp 37–42

  • Sherstinsky A (2020) Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network. Phys. D 404:132306

    MathSciNet  MATH  Google Scholar 

  • Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. In: Proceedings of the 27th international conference on neural information processing systems, pp 568–576

  • Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    MathSciNet  MATH  Google Scholar 

  • Tahmina Z, Scully PJ, Niels P, Casson AJ, Ozanyan KB (2019) Design and implementation of a convolutional neural network on an edge computing smartphone for human activity recognition. IEEE Access 7:133509–133520

    Google Scholar 

  • Vepakomma P, De D, Das SK, Bhansali S (2015) A-wristocracy: deep learning on wrist-worn sensing for recognition of user complex activities. In: 2015 IEEE 12th International conference on wearable and implantable body sensor networks (BSN). IEEE, pp 1–6

  • Vu TH, Dang A, Dung L, Wang J-C (2017) Self-gated recurrent neural networks for human activity recognition on wearable devices. In: Proceedings of the on thematic workshops of ACM multimedia 2017, pp 179–185

  • Wan S, Qi L, Xiaolong X, Tong C, Zonghua G (2020) Deep learning models for real-time human activity recognition with smartphones. Mobile Netw Appl 25(2):743–755

    Google Scholar 

  • Wang Z, Jiang M, Yaohua H, Li H (2012) An incremental learning method based on probabilistic neural networks and adjustable fuzzy clustering for human activity recognition by using wearable sensors. IEEE Trans Inf Technol Biomed 16(4):691–699

    Google Scholar 

  • Wang Z, Yan W, Oates T (2017) Time series classification from scratch with deep neural networks: a strong baseline. In: 2017 international joint conference on neural networks (IJCNN). IEEE, pp 1578–1585

  • Xia K, Huang J, Wang H (2020) LSTM-CNN architecture for human activity recognition. IEEE Access 8:56855–56866

    Google Scholar 

  • Xiaokang Z, Wei L, Kevin I, Kai W, Hao W, Yang LT, Qun J (2020) Deep-learning-enhanced human activity recognition for internet of healthcare things. IEEE Internet Things J 7(7):6429–6438

    Google Scholar 

  • Yuan D, Chang X, Huang P-Y, Liu Q, He Z (2020) Self-supervised deep correlation tracking. IEEE Trans Image Process 30:976–985

    Google Scholar 

  • Zebin T, Sperrin M, Peek N, Casson AJ (2018) Human activity recognition from inertial sensor time-series using batch normalized deep lstm recurrent networks. In: 2018 40th annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, pp 1–4

  • Zhou X, Belkin M (2014) Semi-supervised learning. In: Academic press library in signal processing, vol 1, Elsevier, pp 1239–1269

  • Zhu J, Lou X, Ye W (2021) Lightweight deep learning model in mobile edge computing for radar-based human activity recognition. IEEE Int Things J 8(15):12350–12359

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Computer and Information Techololgy, Jordan University of Science and Technology, Irbid, Jordan

    Luay Alawneh, Mahmoud Al-Ayyoub, Ziad A. Al-Sharif & Ahmed Shatnawi

Authors
  1. Luay Alawneh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mahmoud Al-Ayyoub
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ziad A. Al-Sharif
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ahmed Shatnawi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luay Alawneh.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alawneh, L., Al-Ayyoub, M., Al-Sharif, Z.A. et al. Personalized human activity recognition using deep learning and edge-cloud architecture. J Ambient Intell Human Comput 14, 12021–12033 (2023). https://doi.org/10.1007/s12652-022-03752-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-022-03752-w

Keywords

Search

Navigation