Virtual Reality

, Volume 22, Issue 4, pp 339–351 | Cite as

An evaluation of multimodal interaction techniques for 3D layout constraint solver in a desktop-based virtual environment

  • Marounene Kefi
  • Thuong N. HoangEmail author
  • Paul Richard
  • Eulalie Verhulst
Original Article


We propose a new approach to the 3D layout problems based on the integration of constraint programming and virtual reality interaction techniques. Our method uses an open-source constraint solver integrated in a popular 3D game engine. We designed multimodal interaction techniques for the system, based on gesture and voice input. We conducted a user study with an interactive task of laying out room furniture to compare and evaluate the mono- and multimodal interaction techniques. Results showed that voice command provided the best performance and was most preferred by participants, based on the analysis of both objective and subjective data. Results also revealed that there was no significant difference between the voice and multimodal input (voice and gesture). Our original approach opens the way to multidisciplinary theoretical work and promotes the development of high-level applications for the VR applications.


Interaction techniques Constraint solver 3D layout Virtual environments Multimodality 


  1. Azenkot S, Lee NB (2013) Exploring the use of speech input by blind people on mobile devices. In: Proceedings of the 15th international ACM SIGACCESS conference on computers and accessibility. ACM, pp 1–8Google Scholar
  2. Billinghurst M (1998) Put that where? Voice and gesture at the graphics interface. SIGGRAPH Comput. Graph. 32(4):60–63CrossRefGoogle Scholar
  3. Bolt RA, Herranz E (1992) Two-handed gesture in multi-modal natural dialog. In: 5th Annual ACM symposium on user interface software and technology, pp 7–14Google Scholar
  4. Cal C, Baghabra J, Boges DJ, Holst GR, Kreshuk A, Hamprecht FA, Magistretti PJ (2016) Three-dimensional immersive virtual reality for studying cellular compartments in 3D models from EM preparations of neural tissues. J Comp Neurol 524(1):23–38CrossRefGoogle Scholar
  5. Calderon C, Cavazza M, Diaz D (2003) A new approach to the interactive resolution of configuration problems in virtual environments. Lect Notes Comput Sci 2733:112–122CrossRefGoogle Scholar
  6. Chun LM, Arshad H, Piumsomboon T, Billinghurst M (2015) A combination of static and stroke gesture with speech for multimodal interaction in a virtual environment. In: 2015 International conference on electrical engineering and informatics (ICEEI). IEEE, pp 59–64Google Scholar
  7. Fages F, Soliman S, Coolenand R (2004) CLPGUI: a generic graphical user interface for constraint logic programming. Constraints 9:241–262CrossRefGoogle Scholar
  8. Fernando T, Murray N, Tan K, Wimalaratne P (1999) Software architecture for a constraint-based virtual environment. In: Proceedings of the ACM symposium on virtual reality software and technology, pp 147–154Google Scholar
  9. Gao S, Wan H, Peng Q (2000) An approach to solid modeling in a semi-immersive virtual environment. Comput Graph 24:191–202CrossRefGoogle Scholar
  10. Garcia VM, Ruíz MP, Pérez JR (2010) Voice interactive classroom, a service-oriented software architecture for speech-enabled learning. J Netw Comput Appl 33:603–610CrossRefGoogle Scholar
  11. Goel V, Slusky M, van Hoeve WJ, Furman KC, Shao Y (2015) Constraint programming for LNG ship scheduling and inventory management. Eur J Oper Res 241(3):662–673MathSciNetCrossRefGoogle Scholar
  12. Hart S, Staveland L (1988) Development of NASA-TLX (Task Load Index): results of empirical and theoretical research, pp 139–183Google Scholar
  13. Honda K, Mizoguchi F (1995) Constraint-based approach for automatic spatial layout planning. Conference on artificial intelligence for applications. IEEE Press, Ne YorkGoogle Scholar
  14. ISO/IEC (2011) 25010:2011 Systems and software engineering—systems and software quality requirements and evaluation (SQuaRE)—system and software quality models, ISOGoogle Scholar
  15. Jacob R, Sibert L, MacFarlane D, Mullen P Jr (1994) Integrality and separability of input devices. Comput Hum Interact 1:3–26CrossRefGoogle Scholar
  16. Jacquenot G (2009) Méthode générique pour l’optimisation d’agencement géometrique et fonctionnel. Thése de Doctorat, Ecole Centrale de NantesGoogle Scholar
  17. Jaimes A, Sebe N (2007) Multimodal human–computer interaction: a survey. Comput Vis Image Underst 108(1):116–134CrossRefGoogle Scholar
  18. Kefi M, Barichard V, Richard P (2012) A constraint-solver based tool for user-assisted interactive 3D layout. In: ICTAI, pp 199–206Google Scholar
  19. Ku P-Y et al (2013) Ergonomics design on expert convenience of voice-based interface for vehicles AV systems. In: Kurosu M (ed) Human–computer interaction. Applications and services: 15th international conference, HCI international 2013, Las Vegas, NV, USA, July 21–26, 2013, Proceedings, Part II. Springer, Berlin, pp 606–611CrossRefGoogle Scholar
  20. Kulyukin VA (2004) Human–robot interaction through gesture-free spoken dialogueCrossRefGoogle Scholar
  21. Laviola JJ Jr (1999) Whole-hand and speech input in virtual environments (Doctoral dissertation, Brown University)Google Scholar
  22. Medjdoub B (2004) Constraint-based adaptation for complex space configuration in building services. J Inf Technol Constr 243:627–636Google Scholar
  23. Patrick E et al (2000) Using a large projection screen as an alternative to head-mounted displays for virtual environments. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACMGoogle Scholar
  24. Pfefferkorn C (1975) A heuristic problem solving design system for equipment or furniture layouts. Commun ACM 18(5):286–297CrossRefGoogle Scholar
  25. Pires JN (2006) Industrial robots programming: building applications for the factories of the future. Springer, BerlinGoogle Scholar
  26. Régin JC (2004) Modlisation et Contraintes Globales en Programmation par Contraintes. Habilitation diriger des recherches, Universit de NiceGoogle Scholar
  27. Rogalski T, Wielgat R (2010) A concept of voice guided general aviation aircraft. Aerosp Sci Technol 14:321–328CrossRefGoogle Scholar
  28. Rogowski A (2012a) A industrially oriented voice control system. Robot Comput Integr Manuf 28:303–315CrossRefGoogle Scholar
  29. Rogowski A (2012b) Web-based remote voice control of robotized cells. Robot Comput Integr Manuf 4:77–89Google Scholar
  30. Rossi F, Van Beek P, Walsh T (eds) (2006) Handbook of constraint programming. Elsevier, AmsterdamzbMATHGoogle Scholar
  31. Sanchez S, Le Roux O, Inglese F, Luga H, Gaildart V (2003) Constraint-based 3D-object layout using a genetic algorithm. 3IAGoogle Scholar
  32. Schulte C, Tack G, Lagerkvist M (2013) Modeling with GecodeGoogle Scholar
  33. Seth A, Vance JM, Oliver JH (2011) Virtual reality for assembly methods prototyping: a review. Virtual Real 15(1):5–20CrossRefGoogle Scholar
  34. Tim T, Rafael B, Ruben MS, Klaas JK (2009) Rule-based layout solving and its application to procedural interior generation. In: Proceedings of the CASA workshop on 3D advanced media in gaming and simulation (3AMIGAS), pp 212–227Google Scholar
  35. Turk M (2014) Multimodal interaction: a review. Pattern Recogn Lett 36:189–195CrossRefGoogle Scholar
  36. Unity 3D Game Engine (2013)
  37. Vacher M, Lecouteux B, Istrate D, Joubert T, Portet F et al (2013) Experimental evaluation of speech recognition technologies for voice-based home automation control in a smart home. In: 4th Workshop on speech and language processing for assistive technologies, Grenoble, France, pp 99–105Google Scholar
  38. Weyrich M, Drews P (1999) An interactive environment for virtual manufacturing: the virtual workbench. Comput Ind 38:5–15CrossRefGoogle Scholar
  39. Xu K, Stewart J, Fiume E (2002) Constraint-based automatic placement for scene composition. In: Graphics interface proceedings, University of Calgary, pp 25–34Google Scholar
  40. Zhao W, Madhavan V (2005) Integration of voice commands into a virtual reality environment for assembly design. In: Proceedings of the 10th annual international conference on industrial engineering theory, applications & practice, Clearwater Beach, FL, USAGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2018

Authors and Affiliations

  • Marounene Kefi
    • 1
  • Thuong N. Hoang
    • 2
    Email author
  • Paul Richard
    • 1
  • Eulalie Verhulst
    • 1
  1. 1.LARISUniversity of AngersAngersFrance
  2. 2.School of Information TechnologyDeakin UniversityMelbourneAustralia

Personalised recommendations