Skip to main content
SpringerLink
Log in

Crossmodal perception in virtual reality

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

With the proliferation of low-cost, consumer level, head-mounted displays (HMDs) we are witnessing a reappearance of virtual reality. However, there are still important stumbling blocks that hinder the achievable visual quality of the results. Knowledge of human perception in virtual environments can help overcome these limitations. In this work, within the much-studied area of perception in virtual environments, we look into the less explored area of crossmodal perception, that is, the interaction of different senses when perceiving the environment. In particular, we look at the influence of sound on visual perception in a virtual reality scenario. First, we assert the existence of a crossmodal visuo-auditory effect in a VR scenario through two experiments, and find that, similar to what has been reported in conventional displays, our visual perception is affected by auditory stimuli in a VR setup. The crossmodal effect in VR is, however, lower than that present in a conventional display counterpart. Having asserted the effect, a third experiment looks at visuo-auditory crossmodality in the context of material appearance perception. We test different rendering qualities, together with the presence of sound, for a series of materials. The goal of the third experiment is twofold: testing whether known interactions in traditional displays hold in VR, and finding insights that can have practical applications in VR content generation (e.g., by reducing rendering costs).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. The original experiment [49] reported frames in a regular analog screen whose typical framerate is 25 frames per second. Since the framerate of our screen and the HMD (Oculus Rift) were very different, we adjusted the pause to last 1/25 seconds. Therefore, throughout the paper the terminology is as follows: one frame is equivalent to 1/25 seconds, and two frames are equivalent to 2/25 seconds.

References

  1. Allue M, Serrano A, Bedia MG, Masia B (2016) Crossmodal perception in immersive environments. In: Spanish computer graphics conference (CEIG)

  2. Avanzini F, Rocchesso D (2001) Controlling material properties in physical models of sounding objects. In: ICMC

  3. Baughman AK, McCrory NA, Pandey D, Pandey R Augmented reality enabled response modification, Feb. 13 2018. US Patent 9,891,884

  4. Bertenthal BI, Banton T, Bradbury A (1993) Directional bias in the perception of translating patterns. Perception 22(2):193–207

    Article  Google Scholar 

  5. Billger M, d’Elia S Color appearance in virtual reality: a comparison between a full-scale room and a virtual reality simulation. In: 9th Congress of the international color association (2002), International Society for Optics and Photonics, pp 122–126

  6. Bonneel N, Suied C, Viaud-Delmon I, Drettakis G (2010) Bimodal perception of audio-visual material properties for virtual environments. ACM Trans Appl Percept 7(1):1–16

    Article  Google Scholar 

  7. Cunningham D, Wallraven C (2011) Experimental design: from user studies to psychophysics, 1st edn. A. K Peters, Ltd., Natick

    Book  Google Scholar 

  8. Debevec P (1998) Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In: Proceedings of the 25th annual conference on computer graphics and interactive techniques. ACM, pp 189–198

  9. Dimitropoulos K, Manitsaris S, Tsalakanidou F, Nikolopoulos S, Denby B, Al Kork S, Crevier-Buchman L, Pillot-Loiseau C, Adda-Decker M, Dupont S et al (2014) Capturing the intangible an introduction to the i-treasures project. In: 2014 International conference on computer vision theory and applications (VISAPP), vol 2. IEEE, pp 773–781

  10. Doulamis N, Doulamis A, Ioannidis C, Klein M, Ioannides M (2017) Modelling of static and moving objects: digitizing tangible and intangible cultural heritage. In: Mixed reality and gamification for cultural heritage. Springer, pp 567–589

  11. Finnegan DJ, O’Neill E, Proulx MJ (2016) Compensating for distance compression in audiovisual virtual environments using incongruence. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ’16. ACM, New York, pp 200–212

  12. Fleming RW, Dror RO, Adelson EH (2003) Real-world illumination and the perception of surface reflectance properties. J Vis 3(5):3–3

    Article  Google Scholar 

  13. Fodor J (2000) The mind doesn’t work that way: the scope and limits of computational psychology. MIT Press, Cambridge

    Book  Google Scholar 

  14. Fujisaki W, Goda N, Motoyoshi I, Komatsu H, Nishida S (2014) Audiovisual integration in the human perception of materials. J Vis 14(4):12–12

    Article  Google Scholar 

  15. Fujisaki W, Tokita M, Kariya K (2015) Perception of the material properties of wood based on vision, audition, and touch. Vis Res 109:185–200

    Article  Google Scholar 

  16. Giordano BL, McAdams S (2006) Material identification of real impact sounds: effects of size variation in steel, glass, wood, and plexiglass plates. J Acoust Soc Am 119(2):1171–1181

    Article  Google Scholar 

  17. Grassi M (2005) Do we hear size or sound? Balls dropped on plates. Percep Psychophys 67(2):274–284

    Article  Google Scholar 

  18. Grelaud D, Bonneel N, Wimmer M, Asselot M, Drettakis G (2009) Efficient and practical audio-visual rendering for games using crossmodal perception. In: Proceedings of the 2009 symposium on interactive 3D graphics and games, I3D ’09. ACM, New York, pp 177–182

  19. Guttentag DA (2010) Virtual reality: applications and implications for tourism. Tour Manage 31(5):637–651

    Article  Google Scholar 

  20. Hairston DW, Hodges DA, Burdette JH, Wallace MT (2006) Auditory enhancement of visual temporal order judgment. NeuroReport 17(8):791–5

    Article  Google Scholar 

  21. Hoaglin DC, Iglewicz B (1987) Fine-tunning some resistant rules for outlier labeling. J Am Stat Assoc 82(400):1147–1149

    Article  Google Scholar 

  22. Hoeg ER, Gerry LJ, Thomsen L, Nilsson NC, Serafin S (2017) Binaural sound reduces reaction time in a virtual reality search task. In: 2017 IEEE 3rd VR workshop on sonic interactions for virtual environments (SIVE), pp 1–4

  23. Kerr WB, Pellacini F (2010) Toward evaluating material design interface paradigms for novice users. In: ACM SIGGRAPH 2010 Papers, ACM, pp 35:1–35:10

  24. Klatzky RL, Pai DK, Krotkov EP (2000) Perception of material from contact sounds. Presence: Teleoperators Virt Environ 9(4):399–410

    Article  Google Scholar 

  25. Kokkinara E, Slater M (2014) Measuring the effects through time of the influence of visuomotor and visuotactile synchronous stimulation on a virtual body ownership illusion. Perception 43(1):43–58

    Article  Google Scholar 

  26. Koutek CDM, Koutek M Scientific visualization in virtual reality: interaction techniques and application development

  27. Larsen CR, Soerensen JL, Grantcharov TP, Dalsgaard T, Schouenborg L, Ottosen C, Schroeder TV, Ottesen BS (2009) Effect of virtual reality training on laparoscopic surgery: randomised controlled trial. Bmj 338:b1802

    Article  Google Scholar 

  28. Le Van Quyen M (2011) The brainweb of cross-scale interactions. New Ideas Psychol 29:57–63

    Article  Google Scholar 

  29. Maculewicz J, Nilsson NC, Serafin S (2016) An investigation of the effect of immersive visual and auditory feedback on rhythmic walking interaction. In: Proceedings of the audio mostly 2016, AM ’16. ACM, New York, pp 194–201

  30. Martín R, Iseringhausen J, Weinmann M, Hullin MB Multimodal perception of material properties. In: Proceedings of the ACM SIGGRAPH symposium on applied perception (2015). ACM, pp 33–40

  31. Masia B, Wetzstein G, Didyk P, Gutierrez D (2013) A survey on computational displays: pushing the boundaries of optics, computation, and perception. Comput Graph 37(8):1012–1038

    Article  Google Scholar 

  32. Matsumoto K, Ban Y, Narumi T, Yanase Y, Tanikawa T, Hirose M (2016) Unlimited corridor: redirected walking techniques using visuo haptic interaction. In: ACM SIGGRAPH 2016 emerging technologies, SIGGRAPH ’16. ACM, New York, pp 20:1–20:2

  33. McGurk HMJ (1976) Hearing lips and seeing voices. Nature 264:746–8

    Article  Google Scholar 

  34. McNamara A, Mania K, Gutierrez D (2011) Perception in graphics, visualization, virtual environments and animation. SIGGRAPH Asia Courses

  35. Mishra J, Martinez A, Hillyard SA (2013) Audition influences color processing in the sound-induced visual flash illusion. Vis res 93:74–79

    Article  Google Scholar 

  36. Nilsson NC, Suma E, Nordahl R, Bolas M, Serafin S (2016) Estimation of detection thresholds for audiovisual rotation gains. In: 2016 IEEE virtual reality (VR) pp 241–242

  37. Owens A, Isola P, McDermott J, Torralba A, Adelson EH, Freeman WT (2015) Visually indicated sounds. arXiv:1512.08512

  38. Patney A, Salvi M, Kim J, Kaplanyan A, Wyman C, Benty N, Luebke D, Lefohn A (2016) Towards foveated rendering for gaze-tracked virtual reality. ACM Trans Graph (TOG) 35(6):179

    Article  Google Scholar 

  39. Prinz J (2006) Is the mind really modular? In: Stainton RJ (ed) Contemporary debates in cognitive science. Contemporary debates in philosophy. Blackwell Publishing, Malden

  40. Rallis I, Georgoulas I, Doulamis N, Voulodimos A, Terzopoulos P (2017) Extraction of key postures from 3d human motion data for choreography summarization. In: 2017 9th International conference on virtual worlds and games for serious applications (VS-Games). IEEE, pp 94–101

  41. Ramanarayanan G, Ferwerda J, Walter B, Bala K (2007) Visual equivalence: towards a new standard for image fidelity. ACM Trans Graph 26:3

    Article  Google Scholar 

  42. Riecke BE, Väljamäe A, Schulte-Pelkum J (2009) Moving sounds enhance the visually-induced self-motion illusion (circular vection) in virtual reality. ACM Trans Appl Percept 6(2):7:1–7:27

    Article  Google Scholar 

  43. Rojas D, Kapralos B, Cristancho S, Collins K, Hogue A, Conati C, Dubrowski A (2012) Developing effective serious games: the effect of background sound on visual fidelity perception with varying texture resolution. In: MMVR, pp 386–392

  44. Rojas D, Kapralos B, Hogue A, Collins K, Nacke L, Cristancho S, Conati C, Dubrowski A (2013) The effect of sound on visual fidelity perception in stereoscopic 3-d. IEEE Trans Cybern 43(6):1572–1583

    Article  Google Scholar 

  45. Rojas D, Kapralos B, Collins K, Dubrowski A (2014) The effect of contextual sound cues on visual fidelity perception. Stud Health Technol Inf 196:346–352

    Google Scholar 

  46. Rojas D, Cowan B, Kapralos B, Colllins K, Dubrowski A (2015) The effect of sound on visual realism perception and task completion time in a cel-shaded serious gaming virtual environment. In: 2015 Seventh international workshop on quality of multimedia experience (QoMEX). IEEE, pp 1–6

  47. Samuels R (2000) Massively modular minds: evolutionary psychology and cognitive architecture. In: Carruthers P, Chamberlain A (eds) Evolution and the human mind. Cambridge University Press, Cambridge

    Chapter  Google Scholar 

  48. Sekuler R, Sekuler A, Brackett T (1995) When visual objects collide: repulsion and streaming. Investig Ophthalmol Vis Sci 36:50

    Google Scholar 

  49. Sekuler R, Sekuler AB, Lau R (1997) Sound alters visual motion perception. Nature 385(6614):308

    Article  Google Scholar 

  50. Serrano A, Gutierrez D, Myszkowski K, Seidel H-P, Masia B (2016) An intuitive control space for material appearance. ACM Trans Graph (SIGGRAPH ASIA) 2016(35):6

    Google Scholar 

  51. Seth A, Vance JM, Oliver JH (2011) Virtual reality for assembly methods prototyping: a review. Virt Real 15(1):5–20

    Article  Google Scholar 

  52. Seymour NE, Gallagher AG, Roman SA, O’brien MK, Bansal VK, Andersen DK, Satava RM (2002) Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 236(4):458

    Article  Google Scholar 

  53. Shams LKR (2010) Crossmodal influences on visual perception. Physics of Life Reviews

  54. Shams L, Kamitani YSS (2000) What you see is what you hear? Nature 408:788

    Article  Google Scholar 

  55. Shams L, Kamitani Y, Shimojo S (2002) Visual illusion induced by sound. Cogn Brain Res 14:147–152

    Article  Google Scholar 

  56. Shimojo S, Scheier C, Nijhawan R, Shams L, Kamitani Y, Watanabe K (2001) Beyond perceptual modality: auditory effects on visual perception. Acoust Sci Technol 22(2):61–67

    Article  Google Scholar 

  57. Sillion FX, Rushmeier H, Dorsey J (2008) Digital modeling of material appearance. Morgan Kaufmann/Elsevier

  58. Suh K-S, Lee YE (2005) The effects of virtual reality on consumer learning: an empirical investigation. Mis Q, 673–697

    Article  Google Scholar 

  59. Suied C, Bonneel N, Viaud-Delmon I (2008) Integration of auditory and visual information in the recognition of realistic objects. Exp Brain Res 194(1):91

    Article  Google Scholar 

  60. Tononi G, Edelman GM (1998) Consciousness and complexity. Science 282:1846–1851

    Article  Google Scholar 

  61. Väljamäe A, Larsson P, Västfjäll D, Kleiner M (2008) Sound representing self-motion in virtual environments enhances linear vection. Presence: Teleoper Virt Environ 17(1):43–56

    Article  Google Scholar 

  62. Van Krevelen D, Poelman R (2010) A survey of augmented reality technologies, applications and limitations. Int J Virt Real 9(2):1

    Article  Google Scholar 

  63. Vangorp P (2009) Human visual perception of materials in realistic computer graphics. PhD thesis, Department of Computer Science, KU Leuven Celestijnenlaan 200A, 3001 Heverlee. Belgium

  64. Vangorp P, Laurijssen J, Dutré P (2007) The influence of shape on the perception of material reflectance. ACM Trans Graph 26:3

    Article  Google Scholar 

  65. Varela F, Lachaux JP, Rodriguez E, Martinerie J (2001) The brainweb: phase synchronization and large-scale integration. Nat Rev Neurosci 2:229–239

    Article  Google Scholar 

  66. Waltl M, Timmerer C, Hellwagner H (2010) Improving the quality of multimedia experience through sensory effects. Quality of Multimedia Experience (QoMEX)

Download references

Author information

Authors and Affiliations

  1. Universidad de Zaragoza, I3A, Zaragoza, Spain

    S. Malpica, A. Serrano, M. Allue, M. G. Bedia & B. Masia

Authors
  1. S. Malpica
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. Serrano
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. M. Allue
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. M. G. Bedia
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. B. Masia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to S. Malpica.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Data processing in Experiments 1 and 2

We first processed the collected data by rejecting those users with stereo vision problems. In order to do this, we discarded a user if during the training the percentage of successful answers was equal or under 70%. We further processed the data by rejecting outliers. To do this, we first calculated for each participant and for each of the twelve conditions the percentage of bouncing answers over the ten trials. Then we used the first and third quartiles (Q1 and Q3), and the interquartile difference (Qd) to find outliers for each condition [21]. We discarded a condition if it fulfilled any of the following inequalities:

$$ \begin{array}{ll} condition < (Q_{1} - K_{d}*Q_{d}) \\ condition > (Q_{3} + K_{d}*Q_{d}) \end{array} $$
(1)

with Qd = Q3 − Q1 and Kd = 1.5. Additionally, if a participant was marked as an outlier for more than one condition, all the answers of the participant were discarded.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Malpica, S., Serrano, A., Allue, M. et al. Crossmodal perception in virtual reality. Multimed Tools Appl 79, 3311–3331 (2020). https://doi.org/10.1007/s11042-019-7331-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-7331-z

Keywords

Search

Navigation