Description Logics and Rules for Multimodal Situational Awareness in Healthcare

  • Georgios MeditskosEmail author
  • Stefanos Vrochidis
  • Ioannis Kompatsiaris
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10132)


We present a framework for semantic situation understanding and interpretation of multimodal data using Description Logics (DL) and rules. More precisely, we use DL models to formally describe contextualised dependencies among verbal and non-verbal descriptors in multimodal natural language interfaces, while context aggregation, fusion and interpretation is supported by SPARQL rules. Both background knowledge and multimodal data, e.g. language analysis results, facial expressions and gestures recognized from multimedia streams, are captured in terms of OWL 2 ontology axioms, the de facto standard formalism of DL models on the Web, fostering reusability, adaptability and interoperability of the framework. The framework has been applied in the eminent field of healthcare, providing the models for the semantic enrichment and fusion of verbal and non-verbal descriptors in dialogue-based systems.


Multimodal data Ontologies Rules Situation awareness 



This work has been partially supported by the H2020-645012 project “KRISTINA: A Knowledge-Based Information Agent with Social Competence and Human Interaction Capabilities”.


  1. 1.
    Atrey, P.K., Hossain, M.A., El Saddik, A., Kankanhalli, M.S.: Multimodal fusion for multimedia analysis: a survey. Multimedia Syst. 16(6), 345–379 (2010)CrossRefGoogle Scholar
  2. 2.
    Attard, J., Scerri, S., Rivera, I., Handschuh, S.: Ontology-based situation recognition for context-aware systems. In: Proceedings of the 9th International Conference on Semantic Systems, I-SEMANTICS 2013, pp. 113–120. ACM (2013)Google Scholar
  3. 3.
    Baader, F., Calvanese, D., McGuinness, D.L., Nardi, D., Patel-Schneider, P.F. (eds.): The Description Logic Handbook: Theory, Implementation, and Applications. Cambridge University Press, Cambridge (2003)zbMATHGoogle Scholar
  4. 4.
    Baumgartner, N., Gottesheim, W., Mitsch, S., Retschitzegger, W., Schwinger, W.: Beaware! - Situation awareness, the ontology-driven way. Data Knowl. Eng. 69(11), 1181–1193 (2010)CrossRefGoogle Scholar
  5. 5.
    Bettini, C., Brdiczka, O., Henricksen, K., Indulska, J., Nicklas, D., Ranganathan, A., Riboni, D.: A survey of context modelling and reasoning techniques. Pervasive Mob. Comput. 6(2), 161–180 (2010)CrossRefGoogle Scholar
  6. 6.
    Damljanović, D., Agatonović, M., Cunningham, H., Bontcheva, K.: Improving habitability of natural language interfaces for querying ontologies with feedback and clarification dialogues. Web Seman. Sci. Serv. Agents World Wide Web 19, 1–21 (2013)CrossRefGoogle Scholar
  7. 7.
    Denaux, R., Dimitrova, V., Cohn, A.G.: Interacting with ontologies and linked data through controlled natural languages and dialogues. In: Do-Form: Enabling Domain Experts to Use Formalised Reasoning-AISB Convention 2013, pp. 18–20. Society for the Study of Artificial Intelligence (2013)Google Scholar
  8. 8.
    Dourlens, S., Ramdane-Cherif, A., Monacelli, E.: Multi levels semantic architecture for multimodal interaction. Appl. Intell. 38(4), 586–599 (2013)CrossRefGoogle Scholar
  9. 9.
    Eiter, T., Ianni, G., Krennwallner, T., Polleres, A.: Rules and ontologies for the semantic web. In: Baroglio, C., Bonatti, P.A., Małuszyński, J., Marchiori, M., Polleres, A., Schaffert, S. (eds.) Reasoning Web. LNCS, vol. 5224, pp. 1–53. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-85658-0_1 CrossRefGoogle Scholar
  10. 10.
    Grau, B.C., Horrocks, I., Motik, B., Parsia, B., Patel-Schneider, P., Sattler, U.: OWL 2: the next step for OWL. Web Seman. Sci. Serv. Agents World Wide Web 6(4), 309–322 (2008)CrossRefGoogle Scholar
  11. 11.
    Harris, S., Seaborne, A.: SPARQL 1.1 query language, W3C recommendation, 21 March 2013.
  12. 12.
    Horrocks, I., Patel-Schneider, P.F., Boley, H., Tabet, S., Grosof, B., Dean, M.: SWRL: a semantic web rule language combining OWL and RuleML. National Research Council of Canada and Stanford University, Technical report, May 2004Google Scholar
  13. 13.
    Knublauch, H., Hendler, J.A., Idehen, K.: SPIN - overview and motivation. W3C member submission, World Wide Web Consortium, February 2011Google Scholar
  14. 14.
    Kokar, M.M., Matheus, C.J., Baclawski, K.: Ontology-based situation awareness. Inf. Fusion 10(1), 83–98 (2009). Special Issue on High-level Information Fusion and Situation AwarenessCrossRefGoogle Scholar
  15. 15.
    Lahat, D., Adali, T., Jutten, C.: Multimodal data fusion: an overview of methods, challenges, and prospects. Proc. IEEE 103(9), 1449–1477 (2015)CrossRefGoogle Scholar
  16. 16.
    Motik, B., Cuenca Grau, B., Sattler, U.: Structured objects in OWL: representation and reasoning. In: Proceedings of the 17th International Conference on World Wide Web (WWW 2008), pp. 555–564. ACM, New York (2008)Google Scholar
  17. 17.
    Perera, C., Zaslavsky, A., Christen, P., Georgakopoulos, D.: Context aware computing for the internet of things: a survey. IEEE Commun. Surv. Tutorials 16(1), 414–454 (2014)CrossRefGoogle Scholar
  18. 18.
    Perperis, T., Giannakopoulos, T., Makris, A., Kosmopoulos, D.I., Tsekeridou, S., Perantonis, S.J., Theodoridis, S.: Multimodal and ontology-based fusion approaches of audio and visual processing for violence detection in movies. Expert Syst. Appl. 38(11), 14102–14116 (2011)Google Scholar
  19. 19.
    Prokofyev, R., Tonon, A., Luggen, M., Vouilloz, L., Difallah, D.E., Cudré-Mauroux, P.: SANAPHOR: ontology-based coreference resolution. In: Arenas, M., et al. (eds.) ISWC 2015. LNCS, vol. 9366, pp. 458–473. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-25007-6_27 CrossRefGoogle Scholar
  20. 20.
    Schneider, M., Rudolph, S., Sutcliffe, G.: Modeling in OWL 2 without restrictions. In: Proceedings of the 10th International Workshop on OWL: Experiences and Directions, Co-located with 10th Extended Semantic Web Conference (2013)Google Scholar
  21. 21.
    Shaw, R., Troncy, R., Hardman, L.: Lode: linking open descriptions of events. In: 4th Asian Conference on the Semantic Web, Shanghai, China, pp. 153–167 (2009)Google Scholar
  22. 22.
    Sleeman, J., Finin, T.: Type prediction for efficient coreference resolution in heterogeneous semantic graphs. In: 2013 IEEE Seventh International Conference on Semantic Computing (ICSC), pp. 78–85. IEEE (2013)Google Scholar
  23. 23.
    Solanas, A., Patsakis, C., Conti, M., Vlachos, I.S., Ramos, V., Falcone, F., Postolache, O., Pérez-Martínez, P.A., Di Pietro, R., Perrea, D.N., et al.: Smart health: a context-aware health paradigm within smart cities. IEEE Commun. Mag. 52(8), 74–81 (2014)CrossRefGoogle Scholar
  24. 24.
    Sonntag, D.: Ontologies and Adaptivity in Dialogue for Question Answering, vol. 4. IOS Press, Heidelberg (2010)zbMATHGoogle Scholar
  25. 25.
    Wahlster, W.: Dialogue systems go multimodal: the SmartKom experience. In: Wahlster, W. (ed.) SmartKom: Foundations of Multimodal Dialogue Systems, pp. 3–27. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  26. 26.
    Wongpatikaseree, K., Ikeda, M., Buranarach, M., Supnithi, T., Lim, A.O., Tan, Y.: Activity recognition using context-aware infrastructure ontology in smart home domain. In: Knowledge, Information and Creativity Support, pp. 50–57 (2012)Google Scholar
  27. 27.
    Ye, J., Dasiopoulou, S., Stevenson, G., Meditskos, G., Kontopoulos, E., Kompatsiaris, I., Dobson, S.: Semantic web technologies in pervasive computing: a survey and research roadmap. Pervasive Mob. Comput. 23, 1–25 (2015)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Georgios Meditskos
    • 1
    Email author
  • Stefanos Vrochidis
    • 1
  • Ioannis Kompatsiaris
    • 1
  1. 1.Information Technologies Institute, Centre for Research and Technology - HellasThessalonikiGreece

Personalised recommendations