Multimedia Tools and Applications

, Volume 78, Issue 2, pp 2073–2104 | Cite as

Ontology-driven semantic unified modelling for concurrent activity recognition (OSCAR)

  • Muhammad SafyanEmail author
  • Zia Ul Qayyum
  • Sohail Sarwar
  • Raúl García-Castro
  • Mehtab Ahmed


Activity recognition has a vital role in smart home operations. One of the major challenges in object-sensor-based activity recognition is to learn the complete activity model derived from a generic activity model for sequential and parallel activities. Such challenge exists due to erratic degrees of dissimilar activities in which inhabitants perform activities in sequential and interleaved fashion while interacting with different objects. The proposed work focuses on recognizing a complete set of actions (of activity) by exploiting different knowledge engineering techniques, ontology-based temporal formalisms and data driven techniques. Semantic Segmentation has been employed to establish the generic activity model. The spurious semantic segmentation produced by sensor noise or erratic behaviour is removed by Allen’s temporal formalism. Moreover, Tversky’s feature-based similarity has been used to remove the highly similar spurious activities produced as a result of mistaken interactions with wrong home objects. The duration to perform activities varies among inhabitants; such duration intervals are identified dynamically using the proposed model in order to have a complete activity model. A comprehensive set of experiments has been carried out for evaluating the proposed model where the results based upon different metrics assert its effectiveness especially when compared with other contemporary techniques.


Complete activity model Personalized activity model Adaptive system Domain activity ontology Concurrent activity recognition 


Supplementary material

11042_2018_6318_MOESM1_ESM.rar (15.6 mb)
ESM 1 (RAR 15964 kb)


  1. 1.
    Allen JF (1983) Maintaining knowledge about temporal intervals. Commun ACM 26.11:832–843CrossRefGoogle Scholar
  2. 2.
    Allen JF, Ferguson G (1994) Actions and events in interval temporal logic. J Log Comput 4.5:531–579MathSciNetCrossRefGoogle Scholar
  3. 3.
    Azkune G et al (2015) Extending knowledge-driven activity models through data-driven learning techniques. Expert Syst Appl 42.6:3115–3128CrossRefGoogle Scholar
  4. 4.
    Bao L, Intille SS (2004) Activity recognition from user-annotated acceleration data. In International Conference on Pervasive Computing (pp. 1–17). Springer Berlin HeidelbergGoogle Scholar
  5. 5.
    Caragliu A, Del Bo C, Nijkamp P (2011) Smart cities in Europe. J Urban Technol 18(2):65–82CrossRefGoogle Scholar
  6. 6.
    Chen L, Nugent C (2009) Ontology-based activity recognition in intelligent pervasive environments. Int J Web Info Syst 5(4):410–430CrossRefGoogle Scholar
  7. 7.
    Chen L, Nugent CD, Wang H (2012) A knowledge-driven approach to activity recognition in smart homes. IEEE Trans Knowl Data Eng 24(6):961–974CrossRefGoogle Scholar
  8. 8.
    Chen L, Hoey J, Nugent CD, Cook DJ, Yu Z (2012) Sensor-based activity recognition. IEEE Trans Syst Man Cybern Part C Appl Rev 42(6):790–808CrossRefGoogle Scholar
  9. 9.
    Chen L, Nugent C, Okeyo G (2014) An ontology-based hybrid approach to activity modeling for smart homes. IEEE Trans Human-Mach Syst 44(1):92–105CrossRefGoogle Scholar
  10. 10.
    Cook D, Schmitter-Edgecombe M, Crandall A, Sanders C, Thomas B (2009) Collecting and disseminating smart home sensor data in the CASAS project. Proc CHI Workshop Dev Shared Home Behav Datasets Adv HCI Ubiquitous Comput Res: 1–7)Google Scholar
  11. 11.
    Cui J, Liu Y, Xu Y, Zhao H, Zha H (2013) Tracking generic human motion via fusion of low-and high-dimensional approaches. IEEE Trans Syst, Man, Cybernet: Syst 43(4):996–1002CrossRefGoogle Scholar
  12. 12.
    Díaz-Rodríguez N, Cadahía OL, Cuéllar MP, Lilius J, Calvo-Flores MD (2014) Handling real-world context awareness, uncertainty and vagueness in real-time human activity tracking and recognition with a fuzzy ontology-based hybrid method. Sensors 14(10):18131–18171CrossRefGoogle Scholar
  13. 13.
    Gayathri KS, Elias S, Shivashankar S (2015) Composite activity recognition in smart homes using Markov logic network, IEEE Computer SocietyGoogle Scholar
  14. 14.
    Gayathri KS, Elias S, Shivashankar S (2015) Composite activity recognition in smart homes using Markov logic network. Ubiquitous Intelligence and Computing and 2015 IEEE 12th Intl Conf on Autonomic and Trusted Computing and 2015 IEEE 15th Intl Conf on Scalable Computing and Communications and Its Associated Workshops (UIC-ATC-ScalCom), 2015 IEEE 12th Intl Conf on, pp. 46-53. IEEEGoogle Scholar
  15. 15.
    Georis, B., Maziere, M., Bremond, F., and Thonnat, M, (2004) A video interpretation platform applied to bank agency monitoring. Proc 2nd workshop Intell Distrib Surv Syst: 46–50Google Scholar
  16. 16.
    Gu T, Wu Z, Tao X, Pung HK, Lu J 2009, March. epsicar: an emerging patterns-based approach to sequential, interleaved and concurrent activity recognition. Pervasive Comput Commun, 2009. PerCom 2009. IEEE Int Conf (pp. 1–9). IEEEGoogle Scholar
  17. 17.
    Gu T, Wang L, Wu Z, Tao X, Lu J (2011) A pattern mining approach to sensor-based human activity recognition. IEEE Trans Knowl Data Eng 23(9):1359–1372CrossRefGoogle Scholar
  18. 18.
    Gubbi J, Buyya R, Marusic S, Palaniswami M (2013) Internet of things (IoT): a vision, architectural elements, and future directions. Future Gen Comput Syst 29(7):1645–1660CrossRefGoogle Scholar
  19. 19.
    Hakeem A, Shah M (2004) Ontology and taxonomy collaborated framework for meeting classification. Proc Int Conf Pattern Recogn: 219–222Google Scholar
  20. 20.
    Helaoui R, Niepert M, Stuckenschmidt H (2011) Recognizing interleaved and concurrent activities using qualitative and quantitative temporal relationships. Perva Mobile Comput 7.6:660–670CrossRefGoogle Scholar
  21. 21.
    Helaoui R, Niepert M, Stuckenschmidt H (2011) Recognizing interleaved and concurrent activities: A statistical-relational approach. Perva Comput Commun (PerCom), 2011 IEEE Int Conf (pp. 1–9). IEEEGoogle Scholar
  22. 22.
    Hemalatha M, Uma V, Aghila G (2012) Time ontology with reference event based temporal relations (retr). Int J Web Semantic Technol 3.1Google Scholar
  23. 23.
  24. 24.
    Kakde A, Gulhane V (2015) Real time composite user activity modelling using hybrid approach for recognition. In Electrical, Computer and Communication Technologies (ICECCT), 2015 IEEE International Conference on (pp. 1–6). IEEEGoogle Scholar
  25. 25.
    Khan S, Safyan M (2014) Semantic matching in hierarchical ontologies. J King Saud Univ-Comput Inf Sci 26(3):247–257Google Scholar
  26. 26.
    Likavec S, Cena F (2015) Property-based semantic similarity: what counts?. AICGoogle Scholar
  27. 27.
    Liu Y, Zhang X, Cui J, Wu C, Aghajan H, Zha H (2010) Visual analysis of child-adult interactive behaviors in video sequences. 16th Int Conf Virtual Syst Multimed (VSMM) 2010:26–33 IEEECrossRefGoogle Scholar
  28. 28.
    Liu Y, Cui J, Zhao H, Zha H (2012) Fusion of low-and high-dimensional approaches by trackers sampling for generic human motion tracking. Pattern Recogn (ICPR), 2012 21st Int Conf (pp. 898–901). IEEEGoogle Scholar
  29. 29.
    Liu Y, Nie L, Han L, Zhang L, Rosenblum DS (2015) Action2Activity: recognizing complex activities from sensor data. IJCAI: 1617–1623Google Scholar
  30. 30.
    Liu L, Cheng L, Liu Y, Jia Y, Rosenblum DS (2016) Recognizing complex activities by a probabilistic interval-based model. AAAI 30:1266–1272Google Scholar
  31. 31.
    Liu Y, Zheng Y, Liang Y, Liu S, Rosenblum DS (2016) Urban water quality prediction based on multi-task multi-view learningGoogle Scholar
  32. 32.
    Liu Y, Zhang L, Nie L, Yan Y, Rosenblum DS (2016) Fortune teller: predicting your career path. AAAI: 201–207Google Scholar
  33. 33.
    Lu Y, Wei Y, Liu L, Zhong J, Sun L, Liu Y (2017) Towards unsupervised physical activity recognition using smartphone accelerometers. Multimed Tools Appl 76(8):10701–10719CrossRefGoogle Scholar
  34. 34.
    Mckeever S, Ye J, Coyle L, Bleakley C, Dobson S (2010) Activity recognition using temporal evidence theory. J Ambient Intell Smart Environ 2.3:253–269Google Scholar
  35. 35.
    Meditskos G, Kontopoulos E, Kompatsiaris I (2015) ReDef: context-aware recognition of interleaved activities using OWL 2 and defeasible reasoning. In SSN-TC/OrdRing@ ISWC: 31–42Google Scholar
  36. 36.
    Milea V, Frasincar F, Kaymak U (2008) Knowledge engineering in a temporal semantic web context. Web Eng, 2008. ICWE'08. Eighth Int Conf. IEEEGoogle Scholar
  37. 37.
    Nevatia R, Hobbs J, Bolles B (2004) An ontology for video event representation. Proc IEEE Workshop Event Detect Recognit: 119Google Scholar
  38. 38.
    Okeyo G, Chen L, Wang H, Sterritt R (2012) A knowledge-driven approach to composite activity recognition in smart environments ubiquitous computing and ambient intelligence vol 7656 of the series lecture notes in computer science. Springer, Berlin Heidelberg, pp 322–329Google Scholar
  39. 39.
    Online referencing (accessed April 2018)
  40. 40.
    Pantelopoulos A, Bourbakis NG (2010) A survey on wearable sensor-based systems for health monitoring and prognosis. IEEE Trans Syst Man Cybern Part C Appl Rev 40(1):1–12CrossRefGoogle Scholar
  41. 41.
    Poppe R 2010. A survey on vision-based human action recognition. Image and vision Pantelopoulos, A. and Bourbakis, N.G., 2010Google Scholar
  42. 42.
    Riboni D, Bettini C (2011) COSAR: hybrid reasoning for context-aware activity recognition. Pers Ubiquit Comput 15(3):271–289CrossRefGoogle Scholar
  43. 43.
    Riboni D, Pareschi L, Radaelli L, Bettini C (2011) Is ontology-based activity recognition really effective?" Pervasive Computing and Communications Workshops (PERCOM Workshops), IEEE International Conference on. IEEEGoogle Scholar
  44. 44.
    Riboni D, Sztyler T, Civitarese G, Stuckenschmidt H (2016) Unsupervised recognition of interleaved activities of daily living through ontological and probabilistic reasoning. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (pp. 1–12). ACMGoogle Scholar
  45. 45.
    Singla G, Cook DJ, Schmitter-Edgecombe M (2009) Tracking activities in complex settings using smart environment technologies. Int J Biosci Psychiatr Technol (IJBSPT) 1(1):25Google Scholar
  46. 46.
    Tapia EM, Intille SS, Larson K (2004) Activity recognition in the home using simple and ubiquitous sensors. In International conference on pervasive computing (pp. 158–175). Springer, Berlin, HeidelbergGoogle Scholar
  47. 47.
    Van Kasteren T, Alemdar H, Ersoy C (2011) Effective performance metrics for evaluating activity recognition methods. In ARCS 2011 VDEGoogle Scholar
  48. 48.
    Welty C, Fikes R, Makarios S (2006) A reusable ontology for fluent in OWL. FOIS 150:226–236Google Scholar
  49. 49.
    Ye J, Stevenson G, Dobson S (2015) KCAR: a knowledge-driven approach for concurrent activity recognition. Perva Mobile Comput 19:47–70CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Muhammad Safyan
    • 1
    • 2
    Email author
  • Zia Ul Qayyum
    • 2
  • Sohail Sarwar
    • 2
  • Raúl García-Castro
    • 3
  • Mehtab Ahmed
    • 4
  1. 1.Iqra University IslamabadIslamabadPakistan
  2. 2.University of GujaratGujaratPakistan
  3. 3.Ontology Engineering GroupUniversidad Politécnica de MadridMadridSpain
  4. 4.GC UniversityLahorePakistan

Personalised recommendations