Skip to main content
Log in

Multi levels semantic architecture for multimodal interaction

Applied Intelligence Aims and scope Submit manuscript

Abstract

This paper presents a semantic architecture for solving multimodal interaction. Our architecture is based on multi agent systems where agents are purely semantic using ontologies and inference system. Multi levels concepts and behavioural models are taken into account to bring a fast high level reasoning on a big amount of percepts and low level actions. We apply this architecture to make a system aware of different situations in a network like tracking object behaviours of the environment. As a proof of concept, we apply our architecture to an assistant robot helping blind or disabled people to cross a road in a virtual reality environment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Algorithm 1
Algorithm 2
Algorithm 3
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29

References

  1. Goodrich MA, Schultz AC Human-robot interaction: a survey. Found Trends Hum-Comput Interact

  2. Landragin F (2007) Physical, semantic and pragmatics levels for multimodal fusion and fission. In: Seventh international workshop on computational semantics, vol 7, Tilburg, The Netherlands, pp 346–350

    Google Scholar 

  3. http://www.w3.org/TR/soap12-part0

  4. Hiroshi I, Toshiyuki K, Katumi K, Toru I (1999) A robot architecture based on situated modules. IROS

  5. Ioannidou A, Repenning A, Webb DC, Keyser D, Luhn L, Daetwyler C (2010) Mr. Vetro: a collective simulation for teaching health science. I. Comput-Support Collab Learn 5(2):141–166

    Article  Google Scholar 

  6. Mohammad Y, Nishida T (2010) Controlling gaze with an embodied interactive control architecture. Appl Intell 32(2):148–163

    Article  Google Scholar 

  7. Mohammad Y, Nishida T (2010) Using physiological signals to detect natural interactive behaviour. Appl Intell 33(1):79–92

    Article  Google Scholar 

  8. Anderson JR, Lebiere C (eds) (1998) The atomic components of thought. Lawrence Erlbaum, Mahwah

    Google Scholar 

  9. Laird J, Newell A, Rosenbloom PS (1987) SOAR: an architecture for general intelligence. Artif Intell J 33(1):1–64. http://www.soartechnology.com

    Article  MathSciNet  Google Scholar 

  10. Rao AS, Georgeff MP (1991) Modelling rational agents within a BDI-architecture. In: Allen J, Fikes R, Sandells E (eds) Proceedings of the second international conference on principles of knowledge representation and reasoning. Morgan Kaufmann, San Mateo

    Google Scholar 

  11. Landragin F, Denis A, Ricci A, Romary L (2004) Multimodal meaning representation for generic dialogue systems architectures. In: Proc on language resources and evaluation, pp 521–524

    Google Scholar 

  12. Gupta A, Kembhavi A, Davis LS (2009) Observing human-object interactions: using spatial and functional compatibility for recognition. IEEE Trans Pattern Anal Mach Intell 31(10):1775–1789

    Article  Google Scholar 

  13. Wang J, Byrnes J, Valtorta M, Huhns M (2012) On the combination of logical and probabilistic models for information analysis. Appl Intell 36(2):472–497

    Article  Google Scholar 

  14. Sarkar J, Vinh LT, Lee Y-K, Lee S (2011) GPARS: a general-purpose activity recognition system. Appl Intell 35(2):242–259

    Article  Google Scholar 

  15. Coradeschi S, Loutfi A (2008) A review of past and future trends in perceptual anchoring. In: Fritze P (ed) Tools in artificial intelligence. I-Tech Education and Publishing, Vienna

    Google Scholar 

  16. Gruber TR (1993) A translation approach to portable ontology specification. Knowl Acquis 5:199–220

    Article  Google Scholar 

  17. Baumeister J, Reutelshoefer J, Puppe F (2011) KnowWE: a semantic wiki for knowledge engineering. Appl Intell 35(3):323–344

    Article  Google Scholar 

  18. Guarino N (1995) Formal ontology, conceptual analysis and knowledge representation. Int J Hum-Comput Stud 43(5/6):625–640

    Article  Google Scholar 

  19. Giuliani M, Knoll A (2008) MultiML—a general purpose representation language for multimodal human utterances. In: ICMI’08. Chania, Crete, Greece

    Google Scholar 

  20. Johnston M (2009) Building multimodal applications with EMMA. In: ICMI-MLMI’09, Cambridge, MA, USA, 2–4 November 2009. ACM, New York. ISBN 978-1-60558-772-1

    Google Scholar 

  21. Johnston M, Baggia P, Burnett DC, Carter J, Dahl DA, MacCobb G, Ragget D (2009) EMMA: extensible MultiModal annotation markup language. W3C Recommendation, 10 February 2009

  22. Kranstedt A, Kopp S, Wachsmuth I (2002) Murml: A multimodal utterance representation markup language for conversational agents. In: Proc. of the AAMAS. Workshop on “Embodied conversational agents—let’s specify and evaluate them”

    Google Scholar 

  23. Gibbon D, Gut U, Hell B, Looks K, Trippel ATT (2003) A computational model of arm gestures in conversation. In: EUROSPEECH-2003, pp 813–816

    Google Scholar 

  24. Chai J (2002) Semantics-based representation for multimodal interpretation in conversational systems. In: COLING 2002: the 19th international conference on computational linguistics

    Google Scholar 

  25. Sengers P (1998) Narrative intelligence. In: Dautenhahn K (ed) Human cognition and social agent technology, Advances in consciousness, vol. 19. John Benjamins, Amsterdam

    Google Scholar 

  26. Singh P (2005) EM-ONE: an architecture for reflective commonsense thinking. PhD Thesis, MIT

  27. Obrst L (2003) Ontologies for semantically interoperable systems. In: Proceedings of the twelfth international conference on information and knowledge management, New Orleans, LA, USA. ACM Press, New York, pp 366–369. ISBN: 1-58113-723-0

    Chapter  Google Scholar 

  28. Macal CM, North MJ (2006) Tutorial on agent-based modeling and simulation part 2: how to model with agents. In: Perrone LF, Wieland FP, Liu J, Lawson BG, Nicol DM, Fujimoto RM (eds) Proceedings of the 2006 Winter simulation conference

    Google Scholar 

  29. Allan R (2009) Survey of agent based modeling and simulation tools. Technical report. http://epubs.cclrc.ac.uk/work-details?w=50398

  30. Dourlens S, Ramdane-Cherif A (2010) Semantic memory for pervasive architecture. In: ICICA. LNCS, pp 94–102

    Google Scholar 

  31. Dourlens S, Ramdane-Cherif A (2010) Cognitive memory for semantic agents in robotic interaction. In: IEEE ICCI, pp 511–517

    Google Scholar 

  32. Dourlens S, Ramdane-Cherif A (2011) Semantic modeling & understanding of environment behaviours. In: SSCI 2011—IEEE symposium series on computational intelligence, symposium on intelligent agents, Paris, France, 11–15 April

    Google Scholar 

  33. Jensen K (1991) Coloured Petri nets: a high level language for system design and analysis. In: APN 90: proceedings on advances in Petri nets 1990. Springer, New York, pp 342–416

    Chapter  Google Scholar 

  34. Vinh LT, Lee S, Le HX, Ngo HQ, Kim HI, Hanet M Lee Y-K (2011) Semi-Markov conditional random fields for accelerometer-based activity recognition. Appl Intell 35(2):226–241

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Sébastien Dourlens.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Dourlens, S., Ramdane-Cherif, A. & Monacelli, E. Multi levels semantic architecture for multimodal interaction. Appl Intell 38, 586–599 (2013). https://doi.org/10.1007/s10489-012-0387-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-012-0387-3

Keywords

Navigation