Skip to main content
SpringerLink
Log in

Context-based scene recognition from visual data in smart homes: an Information Fusion approach

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

Ambient Intelligence (AmI) aims at the development of computational systems that process data acquired by sensors embedded in the environment to support users in everyday tasks. Visual sensors, however, have been scarcely used in this kind of applications, even though they provide very valuable information about scene objects: position, speed, color, texture, etc. In this paper, we propose a cognitive framework for the implementation of AmI applications based on visual sensor networks. The framework, inspired by the Information Fusion paradigm, combines a priori context knowledge represented with ontologies with real time single camera data to support logic-based high-level local interpretation of the current situation. In addition, the system is able to automatically generate feedback recommendations to adjust data acquisition procedures. Information about recognized situations is eventually collected by a central node to obtain an overall description of the scene and consequently trigger AmI services. We show the extensible and adaptable nature of the approach with a prototype system in a smart home scenario.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. http://www.vividsolutions.com/jts/. Last accessed 7 March 2011.

  2. http://www.cs.rochester.edu/~spark/muri/. Last accessed 7 January 2011.

  3. http://www.giaa.inf.uc3m.es/miembros/jgomez/et/.

  4. Additional information to represent when the recommendation has been created and the starting and ending frames should be added. For the sake of simplicity, we omit this information.

References

  1. Augusto JC (2007) Ambient intelligence: the confluence of ubiquitous/pervasive computing and artificial intelligence. In: Schuster A (ed) Intelligent computing everywhere. Springer, pp 213–234

  2. Ducatel K, Bogdanowicz M, Scapolo F, Leijten J, Burgelman J-C (2001) Scenarios for ambient intelligence in 2010. ftp://ftp.cordis.europa.eu/pub/ist/docs/istagscenarios2010.pdf. Last accessed 3 Jan 2011

  3. Llinas J, Hall DL (2009) Multisensor data fusion. In: Liggins ME, Hall DL, Llinas J (eds) Handbook of multisensor data fusion. CRC Press, pp 1–14

  4. Steinberg AN, Bowman CL (2009) Revisions to the JDL data fusion model. In: Liggins ME, Hall DL, Llinas J (eds) Handbook of multisensor data fusion. CRC Press, pp 45–67

  5. Gómez-Romero J, García J, Kandefer M, Llinas J, Molina JM, Patricio MA, Prentice M, Shapiro SC (2010) Strategies and techniques for use and exploitation of contextual information in high-level fusion architectures. In: Proceedings of the 13th international conference on information fusion (Fusion 2010). Edimburgh, UK

  6. Henricksen K (2003) A framework for context-aware pervasive computing applications. Ph.D. thesis, University of Queensland

  7. Bremond F, Thonnat M (1996) A context representation for surveillance systems. In: Proceedings of the workshop on conceptual descriptions from images at the 4th European Conference on Computer Systems (ECCV’96). Cambridge, UK

  8. Gruber TR (1993) A translation approach to portable ontology specifications. Knowl Acquis 5(2):199–220

    Article  Google Scholar 

  9. Yilmaz A, Javed O, Shah M (2006) Object tracking: a survey. ACM Comput Surv 38(4):1–45

    Article  Google Scholar 

  10. Yang M, Wu Y, Hua G (2009) Context-aware visual tracking. IEEE Trans Pattern Anal Mach Intell 31(7):1195–1209

    Article  Google Scholar 

  11. Vernon D (2008) Cognitive vision: The case for embodied perception. Image Vis Comput 26(1):127–140

    Article  Google Scholar 

  12. Pinz A, Bischof H, Kropatsch W, Schweighofer G, Haxhimusa Y, Opelt A, Ion A (2008) Representations for cognitive vision. Electron Lett Comput Vis Image Anal 7(2):35–61

    Google Scholar 

  13. Nowak C (2003) On ontologies for high-level information fusion. In: Proceedings of the 6th international conference on information fusion (Fusion 2003). Cairns, Australia, pp 657–664

  14. Hitzler P, Krötzsch M, Parsia B, Pater-Schneider PF, Rudolph S (2009) OWL 2 web ontology language primer. http://www.w3.org/TR/owl2-primer/. Last accessed 7 March 2011

  15. Hong J, Suh E, Kim S (2009) Context-aware systems: a literature review and classification. Expert Syst Appl 36:8509–8522

    Article  Google Scholar 

  16. Turaga P, Ivanov YA (2011) Diamond Sentry: integrating sensors and cameras for real-time monitoring of indoor spaces. IEEE Sens J 11(3):593–602

    Article  Google Scholar 

  17. Steinberg AN, Rogova G (2008) Situation and context in data fusion and natural language understanding. In: Proceedings of the 11th international conference on information fusion (Fusion 2008). Cologne, Germany, pp 1–8

  18. Remagnino P, Foresti GL (2009) Computer vision methods for ambient intelligence. Image Vis Comput 27(10):1419–1420

    Article  Google Scholar 

  19. Snidaro L, Foresti GL (2007) Knowledge representation for ambient security. Expert Syst 25(5):321–333

    Article  Google Scholar 

  20. Velastin SA, Boghossian B, Lo B, Sun J, Vicencio-Silva MA (2005) PRISMATICA: Toward Ambient Intelligence in Public Transport Environments. IEEE Trans Syst Man Cybern Part A Syst Hum 35(1):164–182

    Article  Google Scholar 

  21. Vallejo D, Albusac J, Jimenez L, Gonzalez C, Moreno J (2009) A cognitive surveillance system for detecting incorrect traffic behaviors. Expert Syst Appl 36(7):10503–10511

    Article  Google Scholar 

  22. Albusac J, Vallejo D, Castro-Sánchez JJ, Remagnino P, Gonzalez C, Jimenez L (2010) Monitoring complex environments using a knowledge-driven approach based on intelligent agents. IEEE Intell Syst 25(3):24–31

    Article  Google Scholar 

  23. Corchado JM, Bajo J, Abraham A (2008) GerAmi: improving healthcare delivery in geriatric residences. IEEE Intell Syst 23(2):19–25

    Article  Google Scholar 

  24. Neumann B, Möller R (2008) On scene interpretation with description logics. Image Vis Comput 26:82–101

    Article  Google Scholar 

  25. Springer T, Turhan A-Y (2009) Employing description logics in ambient intelligence for modeling and reasoning about complex situations. J Ambient Intell Smart Environ 1(3):235–259

    Google Scholar 

  26. Wu C, Aghajan H (2011) User-centric environment discovery with camera networks in smart homes. IEEE Trans Syst Man Cybern Part A Syst Hum 41(2):375–383

    Article  Google Scholar 

  27. Gómez-Romero J, Patricio MA, García J, Molina JM (2010) Ontology-based context representation and reasoning for object tracking and scene interpretation in video. Expert Syst Appl 38(6):7494–7510

    Article  Google Scholar 

  28. Steinberg AN, Bowman CL (2004) Rethinking the JDL data fusion levels. In: Proceedings of the MSS national symposium on sensor and data fusion. Columbia, SC, USA

  29. Llinas J, Bowman CL, Rogova G, Steinberg AN, Waltz E, White F (2004) Revisiting the JDL data fusion model II. In: Proceedings of the 7th international conference on information fusion (Fusion 2004). Stockholm, Sweden, pp 1218–1230

  30. Gómez-Romero J, García J, Patricio MA, Molina JM (2011) Communication in distributed tracking systems: an ontology-based approach to improve cooperation. Expert Syst (to appear)

  31. Besada JA, García J, Portillo J, Molina JM, Varona A (2005) Airport surface surveillance based on video images. IEEE Trans Aerosp Electron Syst 41(3):1075–1082

    Article  Google Scholar 

  32. Patricio MA, Carbó J, Pérez O, García J, Molina JM (2007) Multi-agent framework in visual sensor networks. EURASIP J Appl Sign Process 1:226–247

    Google Scholar 

  33. Hobbs J, Pan F (2006) Time ontology in OWL. W3C working draft. Available from http://www.w3.org/TR/owl-time. Last accessed 7 March 2011

  34. Noy N, Rector A (2006) Defining n-ary relations on the semantic web. W3C semantic web best practices and deployment working group note. Available from http://www.w3.org/TR/swbp-n-aryRelations. Last accessed 7 March 2011

  35. Gangemi A, Guarino N, Masolo C, Oltramari A, Schneider L (2002) Sweetening ontologies with DOLCE. In: Proceedings of the 13th international conference on knowledge engineering and knowledge management (ECAW02). Sigüenza, Spain, pp 223–233

  36. Maillot N, Thonnat M, Boucher A (2004) Towards ontology-based cognitive vision. Mach Vis Appl 16(1):33–40

    Article  Google Scholar 

  37. Horrocks I, Pater-Schneider PF (2004) A proposal for an OWL rules language. In: Proceedings of the 13th international conference on World Wide Web (WWW 2004). New York, NY, USA, pp 723–731

  38. Motik B, Sattler U (2005) Query answering for OWL-DL with rules. Web Semant Sci Serv Agents World Wide Web 3(1):41–60

    Article  Google Scholar 

  39. Elsenbroich E, Kutz O, Sattler U (2006) A case for abductive reasoning over ontologies. In: Proceedings of the OWL workshop: experiences and directions. Athens, GA, USA

  40. Häarslev V, Möller R (2001) Description of the RACER systems and its applications. In: Proceedings of the international workshop on description logics (DL2001). Stanford University, CA, USA

  41. Gómez-Romero J, García J, Patricio MA, Molina JM, Rogova G (2011) Representation and exploitation of context knowledge in a harbor surveillance scenario. In: Proceedings of the 14th international conference on information fusion (Fusion 2011). Chicago, IL, USA

  42. Katz Y, Cuenca Grau B (2005) Representing qualitative spatial information in OWL-DL. In: Proceedings of OWL: experiences and directions workshop (OWLED 2005). Galway, Ireland

  43. Grütter R, Scharrenback T, Bauer-Messmer B (2008) Improving an RCC-derived geospatial approximation by OWL axioms. In: Proceedings of the 7th International Semantic Web Conference (ISWC 2008). Karlsruhe, Germany, pp 293–306

  44. Randell DA, Cui Z, Cohn AG (1992) A spatial logic based on regions and connection. In: Proceedings of the 3rd international conference on principles of knowledge representation and reasoning. Cambridge, MA, USA, pp 165–176

  45. Renz J (ed) (2002) Qualitative spatial reasoning with topological information. Springer, Berlin

  46. Gómez-Romero J, García J, Patricio MA, Molina JM (2009) Towards the implementation of an ontology-based reasoning system for visual information fusion. In: Proceedings of the 3rd Skövde Workshop on Information Fusion Topics (SWIFT 2009). Skövde, Sweden, pp 5–10

  47. Serrano MA, García J, Patricio MA, Molina JM (2010) Interactive video annotation tool. In: Proceedings of the international symposium on Distributed Computing and Artificial Intelligence (DCAI’10). Salamanca, Spain, pp 325–332

  48. Open Geospatial Consortium (2011) OpenGIS implementation specification for geographic information—simple feature access. Available from http://www.opengeospatial.org/standards/sfa. Last accessed 7 March 2011

  49. Doermann D, Mihalcik D (2000) Tools and techniques for video performance evaluation. In: Proceedings of the 15th international conference on pattern recognition. Barcelona, Spain, pp 167–170

  50. Horridge M, Bechhofer S (2009) The OWL API: a java API for working with OWL 2 ontologies. In: Proceedings of the 6th OWL Experienced and Directions Workshop (OWLED 2009). Chantilly, VA

  51. García J, Molina JM, Besada JA, Portillo JI (2005) A multitarget tracking video system based on fuzzy and neuro-fuzzy techniques. EURASIP J Appl Sign Process 14:2341–2358

    Article  Google Scholar 

  52. Castanedo F (2010) Distributed data fusion in VSNs using multi-agent systems. Ph.D. thesis, University Carlos III of Madrid

Download references

Acknowledgments

This research activity is supported in part by Projects CICYT TIN2008-06742-C02-02/TSI, CICYT TEC2008-06732-C02-02/TEC, CAM CONTEXTS (S2009/TIC-1485) and DPS2008-07029-C02-02.

Author information

Authors and Affiliations

  1. University Carlos III of Madrid, Avd. de la Universidad Carlos III, 22, 28270, Colmenarejo, Madrid, Spain

    Juan Gómez-Romero, Miguel A. Serrano, Miguel A. Patricio, Jesús García & José M. Molina

Authors
  1. Juan Gómez-Romero
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Miguel A. Serrano
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Miguel A. Patricio
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jesús García
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. José M. Molina
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Juan Gómez-Romero.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gómez-Romero, J., Serrano, M.A., Patricio, M.A. et al. Context-based scene recognition from visual data in smart homes: an Information Fusion approach. Pers Ubiquit Comput 16, 835–857 (2012). https://doi.org/10.1007/s00779-011-0450-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-011-0450-9

Keywords

Search

Navigation