Semantic event fusion of computer vision and ambient sensor data for activity recognition to support dementia care

  • Thanos G. Stavropoulos
  • Georgios Meditskos
  • Stelios Andreadis
  • Konstantinos Avgerinakis
  • Katerina Adam
  • Ioannis Kompatsiaris
Original Research


Although many Ambient Intelligence frameworks either address heterogeneous ambient sensing or computer vision techniques, very limited work integrates both techniques in the scope of activity recognition in pervasive environments. This paper presents such a framework that integrates both a computer vision component and heterogeneous sensors with unanimous semantic representation and interpretation, while it also addresses challenges for realistic applications, such as fast, efficient image analysis and ontology-based temporal interpretation models. The framework is validated through an application in clinical dementia assessment yielding positive results and fruitful conclusions for the proposed semantic fusion of vision and sensor observations.


Activity recognition Computer vision Sensors Ambient intelligence Semantic web Ontologies Rules Dementia 


  1. Avgerinakis K, Briassouli A, Kompatsiaris I (2016) Activity detection using sequential statistical boundary detection (ssbd). Comput Vis Image 144:46–61CrossRefGoogle Scholar
  2. Avgerinakis K, Briassouli A, Kompatsiaris I (2013) Recognition of activities of daily living for smart home environments. In: 2013 9th International Conference on Intelligent Environments. IEEE, pp 173–180Google Scholar
  3. Bettini C, Brdiczka O, Henricksen K et al (2010) A survey of context modelling and reasoning techniques. Pervasive Mob Comput 6:161–180.CrossRefGoogle Scholar
  4. Bonner SG (1998) Assisted interactive dwelling house. In: Proc. 3rd TIDE congress: technology for inclusive design and equality improving the quality of life for the European citizen. p 25Google Scholar
  5. Capone A, Barros M, Hrasnica H, Tompros S (2009) A new architecture for reduction of energy consumption of home appliances. In: TOWARDS eENVIRONMENT, European conference of the Czech Presidency of the Council of the EU. pp 1–8Google Scholar
  6. Chang YJ, Chen CH, Lin LF, Han RP, Huang WT, Lee GC (2012) Wireless sensor networks for vital signs monitoring: application in a nursing home. Int J Distrib Sensor Netw. doi: 10.1155/2012/685107
  7. Chatfield K, Lempitsky VS, Vedaldi A, Zisserman A (2011) The devil is in the details: an evaluation of recent feature encoding methods. In: BMVC, vol 2, no 4. p 8Google Scholar
  8. Chen CY, Grauman K (2012) Efficient activity detection with max-subgraph search. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 1274–1281Google Scholar
  9. Chen L, Nugent C (2009) Ontology-based activity recognition in intelligent pervasive environments. Int J Web Inf Syst 5:410–430CrossRefGoogle Scholar
  10. Chen L, Nugent CD, Wang H (2012) A knowledge-driven approach to activity recognition in smart homes. Knowl Data Eng IEEE Trans 24:961–974.CrossRefGoogle Scholar
  11. Dawadi PN, Cook DJ, Schmitter-Edgecombe M, Parsey C (2013) Automated assessment of cognitive health using smart home technologies. Technol Health Care 21(4):323–343Google Scholar
  12. Demongeot J, Virone G, Duchêne F et al (2002) Multi-sensors acquisition, data fusion, knowledge mining and alarm triggering in health smart homes for elderly people. C R Biol 325:673–682CrossRefGoogle Scholar
  13. Derpanis KG, Sizintsev M, Cannons K, Wildes RP (2010) Efficient action spotting based on a spacetime oriented structure representation. In: IEEE conference on computer vision and pattern recognitionGoogle Scholar
  14. De Paola A, Gaglio S, Lo Re G, Ortolani M (2012) Sensor9k: a testbed for designing and experimenting with WSN-based ambient intelligence applications. Pervasive Mob Comput 8:448–466.CrossRefGoogle Scholar
  15. Duchenne O, Laptev I, Sivic J, Bach F (2009) IEEE international conference in computer vision. In: Automatic annotation of human actions in videoGoogle Scholar
  16. Eisenhauer M, Rosengren P, Antolin P (2010) Hydra: a development platform for integrating wireless devices and sensors into ambient intelligence systems. In: The Internet of Things. Springer, pp 367–373Google Scholar
  17. Friedewald M, Costa O Da, Punie Y et al (2005) Perspectives of ambient intelligence in the home environment. Telemat Inf 22:221–238CrossRefGoogle Scholar
  18. Gauzere B, Greco C, Ritrovato P et al (2015) Semantic web technologies for object tracking and video analytics. Lect Notes Comput Sci (Subser Lect Notes Artif Intell Lect Notes Bioinform) 9475:574–585. doi: 10.1007/978-3-319-27863-6_53 Google Scholar
  19. Greco L, Ritrovato P, Saggese A, Vento M (2016) Abnormal event recognition: a hybrid approach using semantic web. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 58–65Google Scholar
  20. Gu T, Wang XH, Pung HK, Zhang DQ (2004) An ontology-based context model in intelligent environments. In: Proceedings of communication networks and distributed systems modeling and simulation conference, pp 270–275Google Scholar
  21. Gámez N, Fuentes L (2011) FamiWare: a family of event-based middleware for ambient intelligence. Pers Ubiquitous Comput 15:329–339.CrossRefGoogle Scholar
  22. Helal S, Mann W, King J et al (2005) The gator tech smart house: a programmable pervasive space. Computer (Long Beach Calif) 38:50–60Google Scholar
  23. Jain M, Van Gemert J, Jegou H et al (2014) Action localization with tubelets from motion. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit 740–747. doi: 10.1109/CVPR.2014.100
  24. Kerssens C, Kumar R, Adams AE, Knott CC, Matalenas L, Sanford JA, Rogers WA (2015) Personalized technology to support older adults with and without cognitive impairment living at home. Am J Alzheimers Dis Other Demen 30:85–97Google Scholar
  25. Kleinberger T, Becker M, Ras E et al (2007) Ambient intelligence in assisted living: enable elderly people to handle future interfaces. In: Universal access in human–computer interaction. Ambient interaction. Springer, pp 103–112Google Scholar
  26. Kläser A, Marszałek M, Schmid C, Zisserman A (2012) Human focused action localization in video. In: Kutulakos KN (ed) Trends and topics in computer vision. ECCV 2010. Lecture notes in computer science, vol 6553. Springer, BerlinGoogle Scholar
  27. Lan T, Wang Y, Mori G (2011) Discriminative figure-centric models for joint action localization and recognition. In: IEEE International conference in computer visionGoogle Scholar
  28. Laptev I, Patrick P (2007) Retrieving actions in movies. In: IEEE international conference in computer visionGoogle Scholar
  29. Le Gal C, Martin J, Lux A, Crowley JL (2001) Smart office: design of an intelligent environment. IEEE Intell Syst 16:60–66CrossRefGoogle Scholar
  30. Ma S, Zhang J, Ikizler-Cinbis N, Sclaroff S (2013) Action recognition and localization by hierarchical space–time segments. 2013 IEEE Int Conf Comput Vis 2744–2751. doi: 10.1109/ICCV.2013.341
  31. Meditskos G, Dasiopoulou S, Efstathiou V, Kompatsiaris I (2013) Ontology patterns for complex activity modelling. In: International workshop on rules and rule markup languages for the semantic web. Springer, Berlin, pp 144–157Google Scholar
  32. Nevatia R, Hobbs J, Bolles B (2004) An ontology for video event representation. In: Conference on computer vision and pattern recognition workshop, 2004. CVPRW’04. IEEE, pp 119–119Google Scholar
  33. Okeyo G, Chen L, Wang H, Sterritt R (2014) Dynamic sensor data segmentation for real-time knowledge-driven activity recognition. Pervasive Mob Comput 10:155–172.CrossRefGoogle Scholar
  34. Oneata D, Verbeek J, Schmid C (2014) Efficient action localization with approximately normalized fisher vectors. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit 2545–2552. doi: 10.1109/CVPR.2014.326
  35. Patkos T, Chrysakis I, Bikakis A et al (2010) A reasoning framework for ambient intelligence. In: Artificial intelligence: theories, models and applications. Springer, pp 213–222Google Scholar
  36. Prince M, Bryce R, Albanese E, Wimo A, Ribeiro W, Ferri CP (2013) The global prevalence of dementia: a systematic review and metaanalysis. Alzheimers Dement 9(1):63–75CrossRefGoogle Scholar
  37. Riboni D, Bettini C (2011) COSAR: hybrid reasoning for context-aware activity recognition. Pers Ubiquitous Comput 15:271–289CrossRefGoogle Scholar
  38. Riboni D, Pareschi L, Radaelli L, Bettini C (2011) Is ontology-based activity recognition really effective? In: Pervasive computing and communications workshops. IEEE, pp 427–431Google Scholar
  39. Shaw R, Troncy R, Hardman L (2009) Lode: linking open descriptions of events. In: The semantic web. Springer, pp 153–167Google Scholar
  40. Sirin E, Parsia B, Grau BC, Kalyanpur A, Katz Y (2007) Pellet: a practical owl-dl reasoner. J Web Semant 5(2):51–53CrossRefGoogle Scholar
  41. Stavropoulos TG, Gottis K, Vrakas D, Vlahavas I (2013) aWESoME: a web service middleware for ambient intelligence. Expert Syst Appl 40:4380–4392.CrossRefGoogle Scholar
  42. Stavropoulos TG, Kontopoulos E, Bassiliades N et al (2014a) Rule-based approaches for energy savings in an ambient intelligence environment. Pervasive Mob Comput. doi: 10.1016/j.pmcj.2014.05.001
  43. Stavropoulos TG, Meditskos G, Kontopoulos E, Kompatsiaris I (2014b) The DemaWare service-oriented AAL platform for people with dementia. Artif Intell Assist Med (AI-AM/NetMed 2014) 11Google Scholar
  44. Stevenson G, Knox S, Dobson S, Nixon P (2009) Ontonym: a collection of upper ontologies for developing pervasive systems. In: Proceedings of the 1st workshop on context, information and ontologies. ACM, p 9Google Scholar
  45. Stevenson GT, Ye J, Dobson SA, Nixon P (2010) Loc8: a location model and extensible framework for programming with location. IEEE Pervasive Comput 9(1):28–37. doi: 10.1109/MPRV.2009.90
  46. Sun C, Tappen M, Foroosh H (2014) Feature-independent action spotting without human localization, segmentation, or frame-wise tracking. In: IEEE conference on computer vision and pattern recognition, pp 2689–2696Google Scholar
  47. Suzuki R, Otake S, Izutsu T, Yoshida M, Iwaya T (2006) Monitoring daily living activities of elderly people in a nursing home using an infrared motion-detection system. Telemed J Health 12(2):146–155Google Scholar
  48. Tamura T, Togawa T, Ogawa M, Yoda M (1998) Fully automated health monitoring system in the home. Med Eng Phys 20:573–579CrossRefGoogle Scholar
  49. Tran D, Yuan J (2012) Max-margin structured output regression for spatio-temporal action localization. Adv Neural Inf Process Syst 25 359–367Google Scholar
  50. Weiser M (1991) The computer for the 21st century. Sci Am 265:94–104CrossRefGoogle Scholar
  51. Willems G, Becker JH, Tuytelaars T, Van Gool L (2009) Exemplar-based action recognition in video. Br Mach Vis Conf 1–11. doi: 10.5244/C.23.90
  52. Wolf P, Schmidt A, Otte JP et al (2010) openAAL-the open source middleware for ambient-assisted living (AAL). In: AALIANCE conference, Malaga, Spain. pp 1–5Google Scholar
  53. Ye J, Coyle L, Dobson S, Nixon P (2007) Ontology-based models in pervasive computing systems. Knowl Eng Rev 22:315–347CrossRefGoogle Scholar
  54. Yuan J, Liu Z, Research M, Wu Y (2009) Discriminative subvolume search for efficient action detection. 2009 IEEE Conf Comput Vis Pattern Recognit, pp 2442–2449. doi: 10.1109/CVPRW.2009.5206671

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Thanos G. Stavropoulos
    • 1
  • Georgios Meditskos
    • 1
  • Stelios Andreadis
    • 1
  • Konstantinos Avgerinakis
    • 1
  • Katerina Adam
    • 1
  • Ioannis Kompatsiaris
    • 1
  1. 1.Center for Research and Technologies, Information Technologies InstituteHellasGreece

Personalised recommendations