Multisensory Physical Environments for Data Representation

  • Patricia SearchEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9748)


This paper reviews theoretical research and projects in data representation that use different sensory modalities, embodiment, physical objects, and immersive environments. Other topics include the impact of cross-modal perception on data representation and the role audiovisual aesthetics play in the interpretation of data. Research has shown that cross-modal perception enhances sensory stimuli. Sound, touch, gesture, and movement engage the user and create holistic environments that provide multi-dimensional representations of complex data relationships. These data representations include data sculptures, ambient displays, and multisensory environments that use our intuitive abilities to process information from different sensory modalities. By using multiple senses, it is possible to increase the number of variables and relationships that can be represented simultaneously in complex data sets.


Multisensory data representation Cross-modal perception Information aesthetics 


  1. 1.
    Search, P.: New media perspectives for information and data design. In: Fabel, L., Spinillo, C., Tiradentes Souto, V. (eds.) Proceedings of the 7th Information Design International Conference, pp. 222–229. Brazilian Society of Information Design, Brasilia, Brazil (2015)Google Scholar
  2. 2.
    Zhao, J., Vande Moere, A.: Embodiment in data sculpture: a model of the physical visualization of information. In: Proceedings of the Conference on Digital Interactive Media in Entertainment and Arts (DIMEA 2008), pp. 343–350. ACM, Athens, Greece (2008)Google Scholar
  3. 3.
    Zhang, J., Wang, H.: The effect of external representations on numeric tasks. Q. J. Exp. Psychol. 58A(5), 817–838 (2005)CrossRefGoogle Scholar
  4. 4.
    Vande Moere, A., Patel, S.: The physical visualization of information: designing data sculptures in an educational context. In: Huang, M., Nguyen, Q., Zhang, K. (eds.) Visual Information Communication (VINCI 2009), pp. 1–23. Springer, Sydney (2009)CrossRefGoogle Scholar
  5. 5.
    Klemmer, S.R., Hartmann, B., Takayama, L.: How bodies matter: five themes for interaction design. In: Proceedings of Designing Interactive Systems (DIS 2006), pp. 140–148 (2006)Google Scholar
  6. 6.
    Dourish, P.: Where the Action Is: The Foundations of Embodied Interaction. MIT Press, Cambridge (2001)Google Scholar
  7. 7.
    Palmerius, K.L., Forsell, C.: The impact of feedback design in haptic volume visualization. In: Third Joint EuroHaptics Conference 2009 and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, World Haptics 2009, pp. 154–159. IEEE Press, New York (2009)Google Scholar
  8. 8.
    Wisneski, C., Ishii, H., Dahley, A., Gorbet, M., Brave, S., Ullmer, B., Yarin, P.: Ambient displays: turning architectural space into an interface between people and digital information. In: Yuan, F., Konomi, S., Burkhardt, H.-J. (eds.) CoBuild 1998. LNCS, vol. 1370, pp. 22–32. Springer, Heidelberg (1998)CrossRefGoogle Scholar
  9. 9.
    Martialay, M.: Immersive experience: The Campfire. The Approach: Discovery, Innovation, and Imagination at Rensselaer Polytechnic Institute (November 18) (2015).
  10. 10.
    Search, P.: The metastructural dynamics of interactive electronic design. Visible Lang. Cult. Dimensions Vis. Commun. 37(2), 146–165 (2003)Google Scholar
  11. 11.
    Berkeley, G.: A New Theory of Vision and Other Writings. E. P. Dutton, New York (1922)Google Scholar
  12. 12.
    Kirsh, D., Maglio, P.: On distinguishing epistemic from pragmatic actions. Cogn. Sci. 18(4), 513–549 (1994)CrossRefGoogle Scholar
  13. 13.
    Hollan, J., Hutchins, E., Kirsh, D.: Distributed cognition: toward a new foundation for human-computer interaction research. ACM Trans. Comput. Hum. Interact. 7(2), 174–196 (2000)CrossRefGoogle Scholar
  14. 14.
    Fish, J., Scrivener, S.: Amplifying the mind’s eye: sketching and visual cognition. Leonardo 23(1), 117–126 (1990)CrossRefGoogle Scholar
  15. 15.
    Fish, J.: The Cognitive Functions of the Sketch. Cheltenham & Gloucester College of Higher Education, CAD Centre internal Report, Cheltenham (1993)Google Scholar
  16. 16.
    Klatzky, R.L.: Allocentric and egocentric spatial representations: definitions, distinctions, and inter-connections. In: Freksa, C., Habel, C., Wender, K.F. (eds.) Spatial Cognition: An Interdisciplinary Approach to Representation and Processing of Spatial Knowledge. LNAI, vol. 1404, pp. 1–17. Springer, Berlin (1998)CrossRefGoogle Scholar
  17. 17.
    Piaget, J.: The Origins of Intelligence in Children. International University Press, New York (1952)CrossRefGoogle Scholar
  18. 18.
    Lakoff, G., Johnson, M.L.: Metaphors We Live By. The University of Chicago Press, Chicago (1980)Google Scholar
  19. 19.
    Abrahamson, D., Lindgren, R.: Embodiment and embodied design. In: Sawyer, R.K. (ed.) The Cambridge Handbook of the Learning Sciences, 2nd edn, pp. 357–376. Cambridge University Press, Cambridge (2014)Google Scholar
  20. 20.
    LeBaron, C., Streeck, J.: Gestures, knowledge, and the world. In: McNeill, D. (ed.) Language and Gesture, pp. 118–138. Cambridge University Press, Cambridge (2000)CrossRefGoogle Scholar
  21. 21.
    Condillac, E.: An Essay on the Origin of Human Knowledge, Being a Supplement to Mr. Locke’s Essay on the Human Understanding. J. Noursse, London (1746)Google Scholar
  22. 22.
    Wang, Q., Nass, C.: Less visible and wireless: two experiments on the effects of microphone type on users’ performance and perception. In: Proceedings of the 23rd ACM SIGCHI Human Factors in Computing Systems, Portland, Oregon, pp. 809–818 (2005)Google Scholar
  23. 23.
    Goldin-Meadow, S., Nusbaum, H., Delly, S.D., Wagner, S.: Explaining math: gesturing lightens the load. Psychol. Sci. 12(6), 516–522 (1991)CrossRefGoogle Scholar
  24. 24.
    Goldin-Meadow, S., Beilock, S.L.: Action’s influence on thought: the case of gesture. Perspect. Psychol. Sci. 5(6), 664–674 (2010)CrossRefGoogle Scholar
  25. 25.
    Kirsch, D.: Embodied cognition and the magical future of interaction design. In: Marshall, P., Antle, A.N., Hoven, E.V.D., Rogers, Y. (eds.) The Theory and Practice of Embodied Interaction HCI and Interaction Design (special issue). ACM Trans. Hum. Comput. Interact. 20(1), pp. 1–30 (2013)Google Scholar
  26. 26.
    Sams, M., Imada, T.: Integration of auditory and visual information in the human brain: neuromagnetic evidence. Soc. Neurosci. Abs. 23, 1305 (1997)Google Scholar
  27. 27.
    Driver, J., Noesselt, T.: Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments. Neuron 57(1), 11–23 (2008)CrossRefGoogle Scholar
  28. 28.
    Beer, A., Watanabe, T.: Specificity of auditory-guided visual perceptual learning suggests crossmodal plasticity in early visual cortex. Exp. Brain Res. 198(2), 353–361 (2009)CrossRefGoogle Scholar
  29. 29.
    Röder, B., Stock, O., Bien, S., Neville, H., Rösler, F.: Speech processing activates visual cortex in congenitally blind humans. Eur. J. Neurosci. 16(5), 930–936 (2002)CrossRefGoogle Scholar
  30. 30.
    Poirier, C., Collignon, O., Scheiber, C., Renier, L., Vanlierde, A., Tranduy, D., Veraart, C., De Volder, A.G.: Auditory motion perception activates visual motion areas in early blind subjects. Neuroimage 31(1), 279–285 (2006)CrossRefGoogle Scholar
  31. 31.
    Weeks, R., Horwitz, B., Aziz-Sultan, A., Tian, B., Wessinger, C.M., Cohen, L.G., Hallett, M., Rauschecker, J.P.: A positron emission tomographic study of auditory localization in the congenitally blind. J. Neurosci. 20(4), 2664–2672 (2000)Google Scholar
  32. 32.
    Hershenson, M.: Reaction time as a measure of intersensory facilitation. J. Exp. Psychol. 63(3), 289–293 (1962)CrossRefGoogle Scholar
  33. 33.
    Nickerson, R.S.: Intersensory facilitation of reaction time: energy summation or preparation enhancement? Psychol. Rev. 80(6), 168–173 (1973)CrossRefGoogle Scholar
  34. 34.
    Posner, M.I., Nissen, M., Klein, R.M.: Visual dominance: an information-processing account of its origins and significance. Psychol. Rev. 83(2), 157–171 (1976)CrossRefGoogle Scholar
  35. 35.
    Simon, J.R., Craft, J.L.: Effects of an irrelevant auditory stimulus on visual choice reaction time. J. Exp. Psychol. 86(2), 272–274 (1970)CrossRefGoogle Scholar
  36. 36.
    Freides, D.: Human information processing and sensory modality: cross-modal functions, information complexity, memory, and deficit. Psychol. Bull. 81(5), 284–310 (1974)CrossRefGoogle Scholar
  37. 37.
    Vroomen, J., de Gelder, B.: Sound enhances visual perception: cross-modal effects of auditory organization on vision. Hum. Percept. Perform. 26(5), 1583–1590 (2000)CrossRefGoogle Scholar
  38. 38.
    Mazza, V., Turatto, M., Rossi, M., Umiltà, C.: How automatic are audiovisual links in exogenous spatial attention? Neuropsychologia 45(3), 514–522 (2007)CrossRefGoogle Scholar
  39. 39.
    McDonald, J.J., Teder-Sälejärvi, W.A., Hillyard, S.A.: Involuntary orienting to sound improves visual per-ception. Nature 407(6806), 906–908 (2000)CrossRefGoogle Scholar
  40. 40.
    Spence, C., Driver, J.: Audiovisual links in endogenous covert spatial orienting. Percept. Psychophysics 59(1), 1–22 (1997)CrossRefGoogle Scholar
  41. 41.
    Chen, Y., Yeh, S.: Catch the moment: multisensory enhancement of rapid visual events by sound. Exp. Brain Res. 198(2), 209–219 (2009)CrossRefGoogle Scholar
  42. 42.
    Bertelson, P., Radeau, M.: Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Percept. Psychophysics 29(6), 578–584 (1981)CrossRefGoogle Scholar
  43. 43.
    Vroomen, J., Bertelson, P., de Gelder, B.: A visual influence in the discrimination of auditory location. In: Proceedings of the International Conference on Auditory-Visual Speech Processing (AVSP 1998), pp. 131–135. Causal Productions, Sydney, Australia (1998)Google Scholar
  44. 44.
    Vroomen, J.: Ventriloquism and the nature of the unity assumption. In: Aschersleben, G., Bachmann, T., Müsseler, J. (eds.) Cognitive Contributions to the Perception of Spatial and Temporal Events, pp. 388–394. Elsevier Science, New York (1999)Google Scholar
  45. 45.
    Vroomen, J., Bertelson, P., de Gelder, B.: The ventriloquist effect does not depend on the direction of deliberate visual attention. Percept. Psychophysics 63(4), 651–659 (2001)CrossRefGoogle Scholar
  46. 46.
    Lewald, J., EhrenBee, W.H., Guski, R.: Spatio-temporal constraints for auditory-visual integration. Behav. Brain Res. 121(1–2), 69–79 (2001)CrossRefGoogle Scholar
  47. 47.
    Leward, J., Guski, R.: Cross-modal perceptual integration of spatially and temporary disparate auditory and visual stimuli. Cogn. Brain. Res. 16(3), 468–478 (2003)CrossRefGoogle Scholar
  48. 48.
    Talsma, D., Senkowski, D., Woldorff, M.: Intermodal attention affects the processing of the temporal alignment of audiovisual stimuli. Exp. Brain Res. 198(2–3), 313–328 (2009)CrossRefGoogle Scholar
  49. 49.
    Neumann, O., Niepel, M.: Timing of “perception” and perception of “time”. In: Kärnbach, C., Schröger, E., Müller, H. (eds.) Psychophysics Beyond Sensation: Laws and Invariants of Human Cognition, pp. 245–269. Erlbaum, Mahwah (2004)Google Scholar
  50. 50.
    Jaśkowski, P.: Simple reaction time and perception of temporal order: dissociations and hypotheses. Percept. Mot. Skills 82(3, Pt 1), 707–730 (1996)CrossRefGoogle Scholar
  51. 51.
    Zampini, M., Shore, D.I., Spence, C.: Audiovisual temporal-order judgments. Exp. Brain Res. 152(2), 198–210 (2003)CrossRefGoogle Scholar
  52. 52.
    Zampini, M., Shore, D.I., Spence, C.: Multisensory temporal-order judgments: the role of hemispheric redundancy. Int. J. Psychophysiol. 50(1), 165–180 (2003)CrossRefGoogle Scholar
  53. 53.
    Neumann, O., Koch, R., Niepel, M., Tappe, T.: Reaction time and temporal-order judgment: correspondence or dissociation. Zeitschrift fur Experimentelle und Angewandte Psychologie 39(4), 621–645 (1992)Google Scholar
  54. 54.
    Boenke, L.T., Deliano, M., Ohl, F.W.: Stimulus duration influences perceived simultaneity in audiovisual temporal-order judgment. Exp. Brain Res. 198(2–3), 233–244 (2009)CrossRefGoogle Scholar
  55. 55.
    Lau, A., Vande Moere, A.: Towards a model of information aesthetic visualization. In: Proceedings of the IEEE International Conference on Information Visualisation (IV 2007), Zurich, Switzerland, pp. 87–92 (2007)Google Scholar
  56. 56.
    Sack, W.: Aesthetics of information visualization. In: Lovejoy, M., Paul, C., Vesna, V. (eds.) Context Providers, pp. 123–150. Intellect, Bristol (2007)Google Scholar
  57. 57.
    Vesna, V.: Database Aesthetics: Art in the Age of Information Overflow. University of Minnesota Press, Minneapolis (2007)Google Scholar
  58. 58.
    Purchase, H., Allder, J.A., Carrington, D.: Metrics for graph drawing aesthetics. J. Vis. Lang. Comput. 13(5), 501–516 (2002)CrossRefzbMATHGoogle Scholar
  59. 59.
    Kurosu, M., Kashimura, K.: Apparent usability vs. inherent usability: experimental analysis on the determinants of the apparent usability. In: Conference Companion on Human Factors in Computing Systems, CHI 1995, Denver, Colorado, USA, pp. 292–293 (1995)Google Scholar
  60. 60.
    Ngo, D., Byrne, J.G.: Another look at a model for evaluating interface aesthetics. Int. J. Appl. Math. Comput. Sci. 11(2), 515–535 (2001)zbMATHGoogle Scholar
  61. 61.
    Stasko, J., Catrambone, R., Guzdial, M., McDonald, K.: An evaluation of space-filling information visualizations for depicting hierarchical structures. Int. J. Hum Comput Stud. 53(5), 663–694 (2000)CrossRefzbMATHGoogle Scholar
  62. 62.
    Cawthon, N., Vande Moere, A.: The effect of aesthetic on the usability of data visualization. In: Proceedings of the IEEE International Conference Information Visualization (IV 2007), Zurich, Switzerland, pp. 637–648 (2007)Google Scholar
  63. 63.
    Spence, C., Senkowski, D., Röder, B.: Crossmodal processing. Exp. Brain Res. 198(2), 107–111 (2009)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.Rensselaer Polytechnic InstituteTroyUSA

Personalised recommendations