The Visual Computer

, Volume 24, Issue 11, pp 955–961 | Cite as

Automatic speech grammar generation during conceptual modelling of virtual environments

  • Lode VanackenEmail author
  • Chris Raymaekers
  • Karin Coninx
Original Article


Speech interfaces are becoming more and more popular as a means to interact with virtual environments but the development and integration of these interfaces is usually still ad hoc, especially the speech grammar creation of the speech interface is a process commonly performed by hand. In this paper, we introduce an approach to automatically generate a speech grammar which is generated using semantic information. The semantic information is represented through ontologies and gathered from the conceptual modelling phase of the virtual environment application. The utterances of the user will be resolved using queries onto these ontologies such that the meaning of the utterance can be resolved. For validation purposes we augmented a city park designer with our approach. Informal tests validate our approach, because they reveal that users mainly use words represented in the semantic data, and therefore also words which are incorporated in the automatically generated speech grammar.


User interfaces & interaction techniques Speech interfaces Conceptual modelling 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Cohen, P.R.: The role of natural language in a multimodal interface. In: Proceedings of the Fifth ACM Symposium on User Interface Software and Technology, Monteray, CA, USA, pp. 143–149 (1992) Google Scholar
  2. 2.
    Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., Clow, J.: Quickset: multimodal interaction for simulation set-up and control. In: Proceedings of the Fifth Conference on Applied Natural Language Processing, pp. 20–24. Morgan Kaufmann, San Francisco (1997). CrossRefGoogle Scholar
  3. 3.
    Coninx, K., De Troyer, O., Raymaekers, C., Kleinermann, F.: VR-DeMo: a tool-supported approach facilitating flexible development of virtual environments using conceptual modelling. In: Virtual Concept 2006 (VC 06), Cancun, Mexico (2006) Google Scholar
  4. 4.
    Conti, G., Ucelli, G., De Amicis, R.: “Verba volant scripta manent” a false axiom within virtual environments. A semi-automatic tool for retrieval of semantics understanding for speech-enabled vr applications. Comput. Graph. 30(4), 619–628 (2006) CrossRefGoogle Scholar
  5. 5.
    Corradini, A., Cohen, P.: On the relationships among speech, gestures, and object manipulation in virtual environments: Initial evidence. In: Proceedings of the International CLASS Workshop on Natural, Intelligent and Effective Interaction in Multimodal Dialogue Systems (2002) Google Scholar
  6. 6.
    Cuppens, E., Coninx, K.: Cogenive: Code generation for interactive virtual environments. In: The Future of User Interface Design Tools, Workshop of ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, United States (2005) Google Scholar
  7. 7.
    Gorniak, P., Roy, D.: Probabilistic grounding of situated speech using plan recognition and reference resolution. In: ICMI ’05: Proceedings of the 7th International Conference on Multimodal Interfaces, pp. 138–143 (2005) Google Scholar
  8. 8.
    Goubran, R.A., Wood, C.: Building an application framework for speech and pen input integration in multimodal learning interfaces. In: ICASSP ’96: Proceedings of the Acoustics, Speech, and Signal Processing Conference Proceedings, IEEE International Conference, pp. 3545–3548. IEEE Computer Society, Washington (1996). Google Scholar
  9. 9.
    Irawati, S., Calderón, D., Ko, H.: Semantic 3d object manipulation using object ontology in multimodal interaction framework. In: Proceedings of the 2005 International Conference on Augmented Tele-Existence, pp. 35–39 (2005) Google Scholar
  10. 10.
    Irawati, S., Calderón, D., Ko, H.: Spatial ontology for semantic integration in 3d multimodal interaction framework. In: ACM International Conference on Virtual Reality Continuum and Its Applications VRCIA, pp. 129–135 (2006) Google Scholar
  11. 11.
    Kaiser, E., Olwal, A., McGee, D., Benko, H., Corradini, A., Li, X., Cohen, P., Feiner, S.: Mutual disambiguation of 3d multimodal interaction in augmented and virtual reality. In: ICMI ’03: Proceedings of the 5th International Conference on Multimodal Interfaces, pp. 12–19 (2003) Google Scholar
  12. 12.
    Martínez, J.I.: An intelligent guide for virtual environments with fuzzy queries and flexible management of stories. Ph.D. thesis, Universidad de Murcia (2004) Google Scholar
  13. 13.
    Cernak, M., Sannier, A.: Command speech interface to virtual reality applications. Technical Report, Virtual Reality Applications Center at Iowa State University of Science and Technology (2002) Google Scholar
  14. 14.
    McGlashan, S.: Speech interfaces to virtual reality. In: Proceedings of 2nd International Workshop on Military Applications of Synthetic Environments and Virtual Reality (1995) Google Scholar
  15. 15.
    Muller, J., Krapichler, C., Nguyen, L.S., Hans Englmeier, K., Lang, M.: Speech interaction in virtual reality. In: Roceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 3757–3760 (1998) Google Scholar
  16. 16.
    Otto, K.A.: The semantics of multi-user virtual environments. In: Workshop towards Semantic Virtual Environments (SVE 2005), pp. 35–39 (2005) Google Scholar
  17. 17.
    OWL Web Ontology Language: OWL. January 2008
  18. 18.
    Pfeiffer, T., Latoschik, M.E.: Resolving object references in multimodal dialogues for immersive virtual environments. In: Proceedings of the IEEE VR2004, Chicago, USA, pp. 35–42 (2004) Google Scholar
  19. 19.
    Resource Description Framework (RDF): RDF. January 2008
  20. 20.
    Sharma, R., Zeller, M., Pavlovic, V.I., Huang, T.S., Lo, Z., Chu, S., Zhao, Y., Phillips, J.C., Schulten, K.: Speech/gesture interface to a visual-computing environment. IEEE Comput. Graph. Appl. 20(2), 29–37 (2000) CrossRefGoogle Scholar
  21. 21.
    SPARQL Query Language for RDF: SPARQL. January 2008
  22. 22.
  23. 23.
    WordNet: January 2008

Copyright information

© Springer-Verlag 2008

Authors and Affiliations

  1. 1.Hasselt University—tUL—IBBT, Expertise Centre for Digital MediaDiepenbeekBelgium

Personalised recommendations