Multimedia Tools and Applications

, Volume 75, Issue 1, pp 381–404 | Cite as

VOX system: a semantic embodied conversational agent exploiting linked data

  • Francisco J. Serón
  • Carlos BobedEmail author


In the last few years, the use of ontologies has spread thanks to the irruption of the Semantic Web. They have become a crucial tool in information systems as they explicitly state the meaning of information, making it possible to share it and to achieve higher levels of interoperability. However, being knowledge representation models as they are, other fields can take advantage of their characteristics to extend their capabilities. In particular, in the context of Embodied Conversational Agents, they can be used to provide them with semantic knowledge and, therefore, enhance their intellectual skills. In this paper, we propose an approach to explore the synergies between these technologies. Thus, we have developed a multimodal ECA that exploits the knowledge provided by the Linked Data initiative to help users in their search information tasks. Based on a semantic-guided keyword search, our approach is flexible enough to: 1) deal with different Linked Data repositories and 2) handle different search/knowledge domains in a multilingual way. To illustrate the potential of our approach, we have focused on the case of DBpedia, as it mirrors the information stored in the Wikipedia, providing a semantic entry to it.


Semantic embodied conversational agents Semantic knowledge Semantic Web 



This work has been partly financied by:

  • The Spanish “Dirección General de Investigación, Ministerio de Economía y Competitividad”, contract number: TIN2011-24660/REPLIKANTS.

  • The Spanish “Ministerio de Economía y Competitividad”, CICYT projects TIN2010-21387-C02-02 and TIN2013-46238-C4-4-R.

  • The Spanish “Ministerio de Industria, Energía y Turismo”, contract number: AVANZA TSI-020606-2012-4/CONTSEM.

  • European Commission: ALFA_GAVIOTA DCI-ALA/19.09.01/10/21526/245-654/ALFAIII (2010) 149.

  • European Commission: 519332-LLP-1-2011-1-PT-KA3-KA3NW/SEGAN.

  • The DGA (Aragonese Gobern), projects INNOVA-A1-064/13 and DGA-FSE.

We thank to Guillermo Esteban, Daniel Martínez and Javier Marco Rubio for their collaboration as contracted, in the development of this project. We also want to thank Eduardo Mena for his contributions during the design and development of the SENED module.


  1. 1.
    Androutsopoulos I, Ritchie GD, Thanisch P (1995) Natural language interfaces to databases - An introduction. Nat Lang Eng 1(1):29–81CrossRefGoogle Scholar
  2. 2.
    Baader F, Calvanese D, McGuinness D, Nardi D, Patel-Scheneider P (2003) The Description Logic handbook. Theory, implementation and applications. Cambridge University PressGoogle Scholar
  3. 3.
    Baeza-Yates RA, Ribeiro-Neto B (1999) Modern information retrieval. Addison-WesleyGoogle Scholar
  4. 4.
    Baldassarri S, Cerezo E, Serón FJ (2008) Maxine: A platform for embodied animated agents. Comput Graph 32(3):430–437CrossRefGoogle Scholar
  5. 5.
    Berners-Lee T, Hendler J, Lassila O (2001) The semantic web. Sci Am 284(5):34–43CrossRefGoogle Scholar
  6. 6.
    Berry DC, Butler L.T, de Rosis F (2005) Evaluating a realistic agent in an advice-giving task. Int J Hum Comput Stud 63(3):304–327CrossRefGoogle Scholar
  7. 7.
    Beun R.J, de Vos E, Witteman C (2003) Embodied Conversational Agents: Effects on memory performance and anthropomorphisation. In: Proceedings of the 4th International Workshop on Intelligent Agents (IVA’03), Kloster Irsee (Germany). Springer, pp 315–319Google Scholar
  8. 8.
    Bizer C, Heath T, Berners-Lee T (2009) Linked data - The story so far. Int J Semant Web Inf Syst 5(3):1–22CrossRefGoogle Scholar
  9. 9.
    Bizer C, Lehmann J, Kobilarov G, Auer S, Becker C, Cyganiak R, Hellmann S (2009) DBpedia - A crystallization point for the web of data. Web Semant Sci Serv Agents World Wide Web 7(3): 154–165CrossRefGoogle Scholar
  10. 10.
    Bobed C, Esteban G, Mena E (2013) Enabling keyword search on Linked Data repositories: An ontology-based approach. International Journal of Knowledge-based and Intelligent Engineering Systems 17(1):67–77Google Scholar
  11. 11.
    Breuing A (2010) Improving human-agent conversations by accessing contextual knowledge from Wikipedia. In: Proceedings of the 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT’10), Toronto (Canada). IEEE Computer Society Press, pp 428–431Google Scholar
  12. 12.
    Cassell J (2001) Embodied conversational agents: representation and intelligence in user interfaces. AI Mag 22 (4):67–84Google Scholar
  13. 13.
    Cassell J, Sullivan J, Prevost S, Churchill EF (2000) Embodied conversational agents. MIT PressGoogle Scholar
  14. 14.
    Cerezo E, Baldasarri S, Hupont I, Serón FJ (2008) Affective computing. I-Tech Education and PublishingGoogle Scholar
  15. 15.
    Cimiano P, Kopp S (2010) Accessing the Web of Data through embodied virtual characters. Semantic Web Journal 1(1,2):83–88Google Scholar
  16. 16.
    Cochran WG, Cox GM (1957) Experimental designs, 2nd edn. WileyGoogle Scholar
  17. 17.
    Duckhorn F, Hoffmann R (2012) Using context-free grammars for embedded speech recognition with weighted finite-state transducers. In: Proceedings of the 13th Annual Conference of the International Communication Association (INTERSPEECH’12), Portland (Oregon, USA), pp. 1003–1006. ISCAGoogle Scholar
  18. 18.
    D’Ulizia A, Ferri F, Grifoni P (2010) Generating multimodal grammars for multimodal dialogue processing. IEEE Trans Syst Man Cybern Syst Hum 40(6):1130–1145CrossRefGoogle Scholar
  19. 19.
    García A, Lamsfus C (2005) An e-learning platform to support vocational training centers on digital security training with virtual tutors and graphical spatial metaphores. In: Proceedings of the International Conference on Education (IADAT-e2005), Biarritz (France), IADAT, pp. 117–121.Google Scholar
  20. 20.
    Graesser A, Chipman P, Haynes B, Olney A (2005) AutoTutor: an intelligent tutoring system with mixed-initiative dialogue. IEEE Trans Educ 48(4):612–618CrossRefGoogle Scholar
  21. 21.
    Gruber TR (1993) A translation approach to portable ontology specifications. Knowl Acquis 5(2):199–220CrossRefGoogle Scholar
  22. 22.
    Gruber TR (1995) Toward principles for the design of ontologies used for knowledge sharing. Int J Hum Comput Stud 43(5-6):907–928CrossRefGoogle Scholar
  23. 23.
    Kalwick DJ (2006) Animating facial features & expressions, 2nd edn. ThompsonGoogle Scholar
  24. 24.
    Kim H, Park J, Oh Y, Kim S, Kim B (2012) Voice command recognition for fighter pilots using grammar tree. In: Computer Applications for Database, Education, and Ubiquitous Computing. International Conferences (EL, DTA and UNESST’12), Kangwondo (Korea). Springer, Berlin Heidelberg, pp 116–119Google Scholar
  25. 25.
    Kimura M, Kitamura Y (2006) Embodied Conversational Agent based on Semantic Web. In: Proceedings of the 9th Pacific Rim International Conference on Agent Computing and Multi-Agent Systems (PRIMA’06), Guilin (China). Springer, pp 734–741Google Scholar
  26. 26.
    Kipp M, Kipp KH, Ndiaye A, Gebhard P (2006) Evaluating the tangible interface and virtual characters in the interactive COHIBIT exhibit. In: Proceedings of the 6th International Conference on Intelligent Virtual Agents (IVA’06), Marina Del Rey (California, USA). Springer, pp 434–444Google Scholar
  27. 27.
    Lester J, Towns S, Fitzgerald P (1999) Achieving affective impact: visual emotive communication in lifelike pedagogical agents. Int J Artif Intell Educ 10(3):278–291Google Scholar
  28. 28.
    Li H, Zhang T, Qiu R, Ma L (2012) Grammar-based semi-supervised incremental learning in automatic speech recognition and labeling. Energy Procedia 17, Part B(0):1843–1849CrossRefGoogle Scholar
  29. 29.
    Lopez V, Uren VS, Sabou M, Motta E (2011) Is question answering fit for the Semantic Web?: A survey. Semantic Web journal 2(2):125–155Google Scholar
  30. 30.
    Marsella SC, Johnson WL, LaBore C (2000) Interactive pedagogical drama. In: Proceedings of the 4th International Conference on Autonomous Agents (AGENTS’00), Barcelona (Spain), ACM, pp. 301–308.Google Scholar
  31. 31.
    Marsi E, van Rooden F (2007) Expressing uncertainty with a talking head in a multimodal question-answering system. In: Proceedings of the Workshop on Multimodal Output Generation (MOG’07), Aberdeen (UK), CTIT, pp. 105–116.Google Scholar
  32. 32.
    Mignonneau L, Sommerer C (2005) Designing emotional, metaphoric, natural and intuitive interfaces for interactive art, edutainment and mobile communications. Comput Graph 29(6):837–851CrossRefGoogle Scholar
  33. 33.
    Mori M, MacDorman KF, Kageki N (2012) The uncanny valley [from the field]. IEEE Robot Autom Mag 19(2):98–100CrossRefGoogle Scholar
  34. 34.
    Motik B, Shearer R, Horrocks I (2009) Hypertableau reasoning for description logics. J Artif Intell Res 36(1):165–228MathSciNetzbMATHGoogle Scholar
  35. 35.
    Mulken SV, Andr E (1998) The persona effect: How substantial is it? In: People and Computers XIII: Proceedings of HCI’98, Sheffield (UK). Springer, pp 3–66Google Scholar
  36. 36.
    Nass C, Steuer J, Tauber ER (1994) Computers are social actors. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’94), Boston (Massachusetts, USA), ACM, pp. 72–78.Google Scholar
  37. 37.
    Ortiz A, Aizpurua I, Posada J (2003) Some techniques for avatar support of digital storytelling systems. In: Proceeding of 1st International Conference on Technologies for Interactive Digital Storytelling and Entertainment (TIDSE’03), Darmstadt (Germany). Fraunhofer IRB Verlag, pp 322–327Google Scholar
  38. 38.
    Reeves B (2000) The benefits of interactive online characters. Center for the study of language and information. Stanford UniversityGoogle Scholar
  39. 39.
    Rieger T (2003) Avatar gestures. Journal of WSCG 11(2):379–386MathSciNetGoogle Scholar
  40. 40.
    Serenko A (2008) A model of user adoption of interface agents for email notification. Interact Comput 20(4–5):461–472CrossRefGoogle Scholar
  41. 41.
    Serenko A, Bontis N, Detlor B (2007) End-user adoption of animated interface agents in everyday work applications. Behav Inform Technol 26(2):119–132CrossRefGoogle Scholar
  42. 42.
    Shadbolt N, Hall W, Berners-Lee T (2006) The semantic web revisited. IEEE Intell Syst 21(3):96–101CrossRefGoogle Scholar
  43. 43.
    Sirin E, Parsia B, Grau BC, Kalyanpur A, Katz Y (2007) Pellet: A practical OWL-DL reasoner. Web Semant Sci Serv Agents World Wide Web 5(2):51–53CrossRefGoogle Scholar
  44. 44.
    Snedecor GW, Cochran WG (1989) Statistical methods, 8th edn. WileyGoogle Scholar
  45. 45.
    Völkel M, Krötzsch M, Vrandecic D, Haller H, Studer R (2006) Semantic Wikipedia. In: Proceedings of the 15th International Conference on World Wide Web (WWW’06), Edinburgh (Scotland), ACM, pp. 585–594.Google Scholar
  46. 46.
    Waltinger U, Breuing A, Wachsmuth I (2011) Interfacing virtual agents with collaborative knowledge: Open domain question answering using Wikipedia-based topic models. Proceedings of the 22nd International Joint Conference on Artificial Intelligence (IJCAI’11), Barcelona (Spain). AAAI Press, pp 1896–1902Google Scholar
  47. 47.
    Wilcock G (2012)WikiTalk: A spokenWikipedia-based open-domain knowledge access system. In: Proceedings of the Workshop on Question Answering for Complex Domains (QACD’12), Mumbai (India). The COLING 2012 Organizing Committee, pp. 57–70Google Scholar
  48. 48.
    Yuan X, Chee YS (2005) Design and evaluation of Elva: an embodied tour guide in an interactive virtual art gallery. Comput Animat Virt W 16(2):109–119CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Department of Computer Science & Systems EngineeringUniversity of ZaragozaZaragozaSpain

Personalised recommendations