Abstract
This paper presents a semantic architecture for solving multimodal interaction. Our architecture is based on multi agent systems where agents are purely semantic using ontologies and inference system. Multi levels concepts and behavioural models are taken into account to bring a fast high level reasoning on a big amount of percepts and low level actions. We apply this architecture to make a system aware of different situations in a network like tracking object behaviours of the environment. As a proof of concept, we apply our architecture to an assistant robot helping blind or disabled people to cross a road in a virtual reality environment.
References
Goodrich MA, Schultz AC Human-robot interaction: a survey. Found Trends Hum-Comput Interact
Landragin F (2007) Physical, semantic and pragmatics levels for multimodal fusion and fission. In: Seventh international workshop on computational semantics, vol 7, Tilburg, The Netherlands, pp 346–350
Hiroshi I, Toshiyuki K, Katumi K, Toru I (1999) A robot architecture based on situated modules. IROS
Ioannidou A, Repenning A, Webb DC, Keyser D, Luhn L, Daetwyler C (2010) Mr. Vetro: a collective simulation for teaching health science. I. Comput-Support Collab Learn 5(2):141–166
Mohammad Y, Nishida T (2010) Controlling gaze with an embodied interactive control architecture. Appl Intell 32(2):148–163
Mohammad Y, Nishida T (2010) Using physiological signals to detect natural interactive behaviour. Appl Intell 33(1):79–92
Anderson JR, Lebiere C (eds) (1998) The atomic components of thought. Lawrence Erlbaum, Mahwah
Laird J, Newell A, Rosenbloom PS (1987) SOAR: an architecture for general intelligence. Artif Intell J 33(1):1–64. http://www.soartechnology.com
Rao AS, Georgeff MP (1991) Modelling rational agents within a BDI-architecture. In: Allen J, Fikes R, Sandells E (eds) Proceedings of the second international conference on principles of knowledge representation and reasoning. Morgan Kaufmann, San Mateo
Landragin F, Denis A, Ricci A, Romary L (2004) Multimodal meaning representation for generic dialogue systems architectures. In: Proc on language resources and evaluation, pp 521–524
Gupta A, Kembhavi A, Davis LS (2009) Observing human-object interactions: using spatial and functional compatibility for recognition. IEEE Trans Pattern Anal Mach Intell 31(10):1775–1789
Wang J, Byrnes J, Valtorta M, Huhns M (2012) On the combination of logical and probabilistic models for information analysis. Appl Intell 36(2):472–497
Sarkar J, Vinh LT, Lee Y-K, Lee S (2011) GPARS: a general-purpose activity recognition system. Appl Intell 35(2):242–259
Coradeschi S, Loutfi A (2008) A review of past and future trends in perceptual anchoring. In: Fritze P (ed) Tools in artificial intelligence. I-Tech Education and Publishing, Vienna
Gruber TR (1993) A translation approach to portable ontology specification. Knowl Acquis 5:199–220
Baumeister J, Reutelshoefer J, Puppe F (2011) KnowWE: a semantic wiki for knowledge engineering. Appl Intell 35(3):323–344
Guarino N (1995) Formal ontology, conceptual analysis and knowledge representation. Int J Hum-Comput Stud 43(5/6):625–640
Giuliani M, Knoll A (2008) MultiML—a general purpose representation language for multimodal human utterances. In: ICMI’08. Chania, Crete, Greece
Johnston M (2009) Building multimodal applications with EMMA. In: ICMI-MLMI’09, Cambridge, MA, USA, 2–4 November 2009. ACM, New York. ISBN 978-1-60558-772-1
Johnston M, Baggia P, Burnett DC, Carter J, Dahl DA, MacCobb G, Ragget D (2009) EMMA: extensible MultiModal annotation markup language. W3C Recommendation, 10 February 2009
Kranstedt A, Kopp S, Wachsmuth I (2002) Murml: A multimodal utterance representation markup language for conversational agents. In: Proc. of the AAMAS. Workshop on “Embodied conversational agents—let’s specify and evaluate them”
Gibbon D, Gut U, Hell B, Looks K, Trippel ATT (2003) A computational model of arm gestures in conversation. In: EUROSPEECH-2003, pp 813–816
Chai J (2002) Semantics-based representation for multimodal interpretation in conversational systems. In: COLING 2002: the 19th international conference on computational linguistics
Sengers P (1998) Narrative intelligence. In: Dautenhahn K (ed) Human cognition and social agent technology, Advances in consciousness, vol. 19. John Benjamins, Amsterdam
Singh P (2005) EM-ONE: an architecture for reflective commonsense thinking. PhD Thesis, MIT
Obrst L (2003) Ontologies for semantically interoperable systems. In: Proceedings of the twelfth international conference on information and knowledge management, New Orleans, LA, USA. ACM Press, New York, pp 366–369. ISBN: 1-58113-723-0
Macal CM, North MJ (2006) Tutorial on agent-based modeling and simulation part 2: how to model with agents. In: Perrone LF, Wieland FP, Liu J, Lawson BG, Nicol DM, Fujimoto RM (eds) Proceedings of the 2006 Winter simulation conference
Allan R (2009) Survey of agent based modeling and simulation tools. Technical report. http://epubs.cclrc.ac.uk/work-details?w=50398
Dourlens S, Ramdane-Cherif A (2010) Semantic memory for pervasive architecture. In: ICICA. LNCS, pp 94–102
Dourlens S, Ramdane-Cherif A (2010) Cognitive memory for semantic agents in robotic interaction. In: IEEE ICCI, pp 511–517
Dourlens S, Ramdane-Cherif A (2011) Semantic modeling & understanding of environment behaviours. In: SSCI 2011—IEEE symposium series on computational intelligence, symposium on intelligent agents, Paris, France, 11–15 April
Jensen K (1991) Coloured Petri nets: a high level language for system design and analysis. In: APN 90: proceedings on advances in Petri nets 1990. Springer, New York, pp 342–416
Vinh LT, Lee S, Le HX, Ngo HQ, Kim HI, Hanet M Lee Y-K (2011) Semi-Markov conditional random fields for accelerometer-based activity recognition. Appl Intell 35(2):226–241
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Dourlens, S., Ramdane-Cherif, A. & Monacelli, E. Multi levels semantic architecture for multimodal interaction. Appl Intell 38, 586–599 (2013). https://doi.org/10.1007/s10489-012-0387-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-012-0387-3