Exploring the Dynamic Aesthetics of Interaction Design

  • Patricia SearchEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11583)


In interactive, electronic communication, where information continually changes and users physically interact with objects, there are numerous elements that define the aesthetic experience. Audiovisual design, dynamic semantic relationships, cross-modal perception, physical interaction and movement, cognition, and memory impact the aesthetics of the design. Static information hierarchies give way to audiovisual patterns that present information in parallel, synchronous formats, as well as linear sequences. Diverse sensory stimuli, interaction in hybrid environments that integrate physical and virtual spaces, and social networking lead to the formation of discursive semantic relationships and dynamic perceptual and cognitive networks that also define the design aesthetics of the work.


Aesthetics Interaction design Semiotics Cross-modal perception Embodiment 


  1. 1.
    Kurosu, M., Kashimura, K.: Apparent usability vs. inherent usability: experimental analysis on the determinants of the apparent usability. In: CHI 1995: Conference Companion on Human Factors in Computing Systems, Denver, Colorado, United States, pp. 292–293 (1995)Google Scholar
  2. 2.
    Ngo, D., Byrne, J.G.: Another look at a model for evaluating interface aesthetics. Int. J. Appl. Math. Comput. Sci. 11(2), 515–535 (2001)zbMATHGoogle Scholar
  3. 3.
    Stasko, J., Catrambone, R., Guzdial, M., McDonald, K.: An evaluation of space-filling information visualizations for depicting hierarchical structures. Int. J. Hum. Comput. Stud. 53(5), 663–694 (2000)zbMATHCrossRefGoogle Scholar
  4. 4.
    Cawthon, N., Vande Moere, A.: The effect of aesthetic on the usability of data visualization. In: Proceedings of the IEEE International Conference Information Visualization (IV 2007), pp. 637–648, Zurich, Switzerland (2007)Google Scholar
  5. 5.
    Kant, I.: Critique of Pure Reason (P. Guyer, A. Wood). Cambridge University Press, Cambridge (1998)Google Scholar
  6. 6.
    Baumgarten, A.: Metaphysics: A Critical Translation with Kant’s Elucidations, Selected Notes, and Related Materials (C.D. Fugate, J. Hymers). Bloomsbury Academic, New York (2014)Google Scholar
  7. 7.
    Tuan, Y.-F.: Space and Place. University of Minnesota, Minneapolis (1977)Google Scholar
  8. 8.
    Norman, D.: The Psychology of Everyday Things. Basic Books, New York (1988)Google Scholar
  9. 9.
    Xenakis, I., Arnellos, A., Spyrou, T., Darzentas, J.: Modelling aesthetic judgment: an interactive-semiotic perspective. Cybern. Hum. Knowing 19(3), 25–51 (2012)Google Scholar
  10. 10.
    Reason, M., Jola, C., Kay, R., Reynolds, D., Kauppi, J., Grobras, M., Tohka, J., Pollick, F.: Spectators’ aesthetic experience of sound and movement in dance performance: a transdisciplinary investigation. Psychol. Aesthet. Creat. Arts 10(1), 42–55 (2016)CrossRefGoogle Scholar
  11. 11.
    Vroomen, J., de Gelder, B.: Sound enhances visual perception: cross-modal effects of auditory or-ganization on vision. Hum. Percept. Perform. 26(5), 1583–1590 (2000)CrossRefGoogle Scholar
  12. 12.
    Mazza, V., Turatto, M., Rossi, M., Umiltà, C.: How automatic are audiovisual links in exogenous spatial attention? Neuropsychologia 45(3), 514–522 (2007)CrossRefGoogle Scholar
  13. 13.
    McDonald, J.J., Teder-Sälejärvi, W.A., Hillyard, S.A.: Involuntary orienting to sound improves visual perception. Nature 407(6806), 906–908 (2000)CrossRefGoogle Scholar
  14. 14.
    Spence, C., Driver, J.: Audiovisual links in endogenous covert spatial orienting. Percept. Psychophys. 59(1), 1–22 (1997)CrossRefGoogle Scholar
  15. 15.
    Beer, A., Watanabe, T.: Specificity of auditory-guided visual perceptual learning suggests crossmodal plasticity in early visual cortex. Exp. Brain Res. 198(2), 353–361 (2009)CrossRefGoogle Scholar
  16. 16.
    Stein, B.E., London, N., Wilkinson, L.K., Price, D.P.: Enhancement of perceived visual intensity by auditory stimuli: a psychophysical analysis. J. Cogn. Neurosci. 8(6), 497–506 (1996)CrossRefGoogle Scholar
  17. 17.
    Chen, Y., Yeh, S.: Catch the moment: multisensory enhancement of rapid visual events by sound. Exp. Brain Res. 198(2), 209–219 (2009)CrossRefGoogle Scholar
  18. 18.
    O’Leary, A., Rhodes, G.: Cross-modal effects on visual and auditory object perception. Percept. Psychophys. 35(6), 565–569 (1984)CrossRefGoogle Scholar
  19. 19.
    Search, P.: HyperGlyphs: using design and language to define hypermedia navigation. J. Educ. Multimed. Hypermedia 2(4), 369–380 (1993)Google Scholar
  20. 20.
    Macken-Horarik, M.: Interacting with the multimodal text: reflections on image and verbiage. Vis. Commun. 3(1), 5–26 (2004)CrossRefGoogle Scholar
  21. 21.
    Search, P.: Defining a sense of place in interactive multimedia design. In: Avgerinou, M., Chandler, S. (eds.) Visual Literacy in the 21st Century: Trends, Demands, And Capacities, pp. 143–148. International Visual Literacy Association, Chicago (2011)Google Scholar
  22. 22.
    Search, P.: The spatial grammar of interaction design: weaving a tapestry of space and time in multimedia computing. In: Griffin, R., Cowden, B., Avgerinou, M. (eds.) Animating the Mind’s Eye, pp. 184–190. International Visual Literacy Association, Loretto (2006)Google Scholar
  23. 23.
    Djajadiningrat, J., Matthews, B., Stienstra, M.: Easy doesn’t do it: skill and expression in tangible aesthetics. Pers. Ubiquitous Comput. 11(8), 657–676 (2007)CrossRefGoogle Scholar
  24. 24.
    Bertelson, P., Radeau, M.: Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Percept. Psychophys. 29(6), 578–584 (1981)CrossRefGoogle Scholar
  25. 25.
    Vroomen, J., Bertelson, P., de Gelder, B.: A visual influence in the discrimination of auditory location. In: Proceedings of the International Conference on Auditory-Visual Speech Processing (AVSP 1998), pp. 131–135. Causal Productions, Sydney, Australia (1998)Google Scholar
  26. 26.
    Vroomen, J.: Ventriloquism and the nature of the unity assumption. In: Aschersleben, G., Bachmann, T., Müsseler, J. (eds.) Cognitive Contributions to the Perception of Spatial and Temporal Events, pp. 388–394. Elsevier Science, New York (1999)Google Scholar
  27. 27.
    Vroomen, J., Bertelson, P., de Gelder, B.: The ventriloquist effect does not depend on the direction of deliberate visual attention. Percept. Psychophys. 63(4), 651–659 (2001)CrossRefGoogle Scholar
  28. 28.
    Lewald, J., EhrenBee, W.H., Guski, R.: Spatio-temporal constraints for auditory-visual integration. Behav. Brain Res. 121(1–2), 69–79 (2001)CrossRefGoogle Scholar
  29. 29.
    Leward, J., Guski, R.: Cross-modal perceptual integration of spatially and temporary disparate auditory and visual stimuli. Cogn. Brain Res. 16(3), 468–478 (2003)CrossRefGoogle Scholar
  30. 30.
    Neumann, O., Niepel, M.: Timing of “perception” and perception of “time”. In: Kärnbach, C., Schröger, E., Müller, H. (eds.) Psychophysics Beyond Sensation: Laws and Invariants of Human Cognition, pp. 245–269. Erlbaum, Mahwah (2004)Google Scholar
  31. 31.
    Jaśkowski, P.: Simple reaction time and perception of temporal order: dissociations and hypotheses. Percept. Motor Skills 82(3, Pt 1), 707–730 (1996)CrossRefGoogle Scholar
  32. 32.
    Zampini, M., Shore, D.I., Spence, C.: Audiovisual temporal-order judgments. Exp. Brain Res. 152(2), 198–210 (2003)CrossRefGoogle Scholar
  33. 33.
    Zampini, M., Shore, D.I., Spence, C.: Multisensory temporal-order judgments: the role of hemispheric redundancy. Int. J. Psychophysiol. 50(1), 165–180 (2003)CrossRefGoogle Scholar
  34. 34.
    Neumann, O., Koch, R., Niepel, M., Tappe, T.: Reaction time and temporal-order judgment: correspondence or dissociation. Z. Exp. Angew. Psychol. 39(4), 621–645 (1992)Google Scholar
  35. 35.
    Boenke, L.T., Deliano, M., Ohl, F.W.: Stimulus duration influences perceived simultaneity in audiovisual temporal-order judgment. Exp. Brain Res. 198(2–3), 233–244 (2009)CrossRefGoogle Scholar
  36. 36.
    Monge, P., Kalmann, M.: Sequentiality, simultaneity, and synchronicity in human communication. In: Watt, J., VanLear, C.A. (eds.) Dynamic Patterns in Communication Processes, pp. 71–91. Sage Publications, Thousand Oaks (1996)Google Scholar
  37. 37.
    Ross, C.: Augmented reality art: a matter of (non)destination. In: Proceedings of the Digital Arts and Culture Conference, University of California, Irvine (2009).
  38. 38.
    Search, P.: Information design opportunities with augmented reality applications. Inf. Des. J. 22(3), 237–246 (2016)Google Scholar
  39. 39.
    Search, P.: Relational aesthetics in interactive art and design. In: Aesthetics@Media, Arts and Culture, pp. 632–643. International Association of Empirical Aesthetics, Taipei (2012)Google Scholar
  40. 40.
    Search, P.: Using new media art and multisensory design for information and data representation. In: Tiradentes Souto, V., Spinillo, C., Portugal, C., Fadel, L. (eds.) Selected Readings of the 7th Information Design International Conference, pp. 179–190. The Brazilian Society of Information Design, Brasilia (2016)Google Scholar
  41. 41.
    Search, P.: Multisensory physical environments for data representation. In: Marcus, A. (ed.) Proceedings of HCI International 2016: Design, user experience, and usability, Part III, pp. 202–213. Springer, Cham, Switzerland (2016)CrossRefGoogle Scholar
  42. 42.
    Norman, D.A.: Introduction to this special section on beauty, goodness, and usability. Hum. Comput. Interact. 19(4), 311–318 (2004)CrossRefGoogle Scholar
  43. 43.
    Brunel, L., Carvalho, P., Goldstone, R.: It does belong together: cross-modal correspondences influence cross-modal integration during perceptual learning. Front. Psychol. 6, 358 (2015). Scholar
  44. 44.
    MacLeod, A., Summerfield, Q.: A procedure for measuring auditory and audiovisual speech-reception thresholds for sentences in noise: rationale, evaluation, and recommendations for use. Br. J. Audiol. 24(1), 29–43 (1990)CrossRefGoogle Scholar
  45. 45.
    Stein, B.E., Meredith, M.A.: The Merging of Senses. MIT Press, Cambridge (1993)Google Scholar
  46. 46.
    Chen, Y.-C., Spence, C.: When hearing the bark helps to identify the dog: semantically-congruent sounds modulate the identification of masked pictures. Cognition 114(3), 389–404 (2010)CrossRefGoogle Scholar
  47. 47.
    Molholm, S., Ritter, W., Murray, M.M., Javitt, D.C., Schroeder, C.E., Foxe, J.J.: Multi-sensory auditory–visual interactions during early sensory processing in humans: a high-density electrical mapping study. Cogn. Brain. Res. 14(1), 115–128 (2002)CrossRefGoogle Scholar
  48. 48.
    Brunel, L., Labeye, E., Lesourd, M., Versace, R.: The sensory nature of episodic memory: sensory priming effects due to memory trace activation. J. Exp. Psychol. Learn. Mem. Cogn. 35(4), 1081–1088 (2009)CrossRefGoogle Scholar
  49. 49.
    Brunel, L., Lesourd, M., Labeye, E., Versace, R.: The sensory nature of knowledge: sensory priming effects in semantic categorization. Q. J. Exp. Psychol. 63(5), 955–964 (2010)CrossRefGoogle Scholar
  50. 50.
    Brunel, L., Goldstone, R.L., Vallet, G., Riou, B., Versace, R.: When seeing a dog activates the bark: multisensory generalization and distinctiveness effects. Exp. Psychol. 60(2), 100–112 (2013)CrossRefGoogle Scholar
  51. 51.
    Connolly, K.: Multisensory perception as an associative learning process. Front. Psychol. 5, 1095 (2014). Scholar
  52. 52.
    Goldstone, R.L., Landy, D., Brunel, L.C.: Improving the perception to make distant con-nections closer. Front. Psychol. 2, 385 (2013). Scholar
  53. 53.
    Search, P.: HyperGlyphs: new multiliteracy models for interactive computing. In: Griffin, R., Lee, J., Williams, V. (eds.) Visual Literacy in Message Design, pp. 171–177. International Visual Literacy Association, Loretto (2002)Google Scholar
  54. 54.
    Merleau-Ponty, M.: Phenomenology of Perception (C. Smith). Routledge, London (1945, repr. 2005)Google Scholar
  55. 55.
    Klemmer, S.R., Hartmann, B., Takayama, L.: How bodies matter: five themes for interaction design. In: Proceedings of Designing Interactive Systems (DIS 2006), pp. 140–148 (2006)Google Scholar
  56. 56.
    Lakoff, G., Johnson, M.L.: Metaphors We Live By. University of Chicago Press, Chicago (1980)Google Scholar
  57. 57.
    Abrahamson, D., Lindgren, R.: Embodiment and embodied design. In: Sawyer, R.K. (ed.) The Cambridge Handbook of the Learning Sciences, 2nd edn, pp. 357–376. Cambridge University Press, Cambridge (2014)Google Scholar
  58. 58.
    Dourish, P.: Where the Action Is: The Foundations of Embodied Interaction. MIT Press, Cambridge (2001)CrossRefGoogle Scholar
  59. 59.
    Palmerius, K.L., Forsell, C.: The impact of feedback design in haptic volume visualization. In: Third Joint EuroHaptics Conference 2009 and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, World Haptics 2009, pp. 154–159. IEEE Press, New York (2009)Google Scholar
  60. 60.
    Penny, S.: Representation, enaction, and the ethics of simulation. In: Wardrip-Fruin, N., Harrigan, P. (eds.) First Person: New Media as Story, Performance, and Game, pp. 73–84. MIT Press, Cambridge (2004)Google Scholar
  61. 61.
    Piaget, J.: Recherches sur l’Abstraction Reflechissante, Vol. I & II. Presses Universitaires de France, Paris (1977)Google Scholar
  62. 62.
    Hopfield, J.J.: Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. 79(8), 2554–2558 (1982)MathSciNetzbMATHCrossRefGoogle Scholar
  63. 63.
    Rumelhart, D.E., Todd, P.M.: Learning and connectionist representations. In: Meyer, D.E., Kornblum, S. (eds.) Attention and Performance XIV: Synergies in Experimental Psychology, Artificial Intelligence, and Cognitive Neuroscience, pp. 3–30. MIT Press, Cambridge (1993)Google Scholar
  64. 64.
    Jones, M., Willits, J., Dennis, S.: Models of semantic memory. In: Busemeyer, J., Wang, Z., Townsend, J., Eidels, A. (eds.) The Oxford Handbook of Computational and Mathematical Psychology, pp. 232–254. Oxford University Press, New York (2015)Google Scholar
  65. 65.
    Tversky, B.: Cognitive maps, cognitive collages, and spatial mental models. In: Frank, A., Campari, I. (eds.) Spatial Information Theory: A Theoretical Basis for GIS, pp. 14–24. Springer, Heidelberg (1993)CrossRefGoogle Scholar
  66. 66.
    Turkle, S., Papert, S.: Epistemological pluralism and the evaluations of the concrete. In: Harel, I., Papert, S. (eds.) Constructivism, pp. 161–192. Ablex Publishing Company, Norwood (1993)Google Scholar
  67. 67.
    Rogers, T.T., McClelland, J.L.: Semantic Cognition. MIT Press, Cambridge (2006)Google Scholar
  68. 68.
    Minsky, M.: The Society of Mind. Simon and Shuster, New York (1988)Google Scholar
  69. 69.
    Bergman, A.: Auditory Scene Analysis. MIT Press, Cambridge (1994)Google Scholar
  70. 70.
    Piaget, J.: The Moral Judgement of the Child. Kegen Paul, Trench, Trubner & Co, London (1932)Google Scholar
  71. 71.
    Moriarty, S.E.: Abduction: a theory of visual interpretation. Commun. Theory 6(2), 167–187 (1996)CrossRefGoogle Scholar
  72. 72.
    Meister, M.L.R., Buffalo, E.A.: Memory. In: Conn, M. (ed.) Conn’s Translational Neuroscience, pp. 693–708. Academic Press, Cambridge (2016)Google Scholar
  73. 73.
    Costello, B.: Rhythms of kinesthetic empathy. Leonardo 47(3), 258–259 (2014)CrossRefGoogle Scholar
  74. 74.
    Search, P.: A new visual literacy discourse for interactive electronic communication. In: Avgerinou, M., Griffin, R., Giesen, J., Search, P. (eds.) Engaging Creativity and Critical Thinking, pp. 181–188. International Visual Literacy Association, Loretto (2009)Google Scholar
  75. 75.
    Ingold, T.: The Perception of the Environment. Routledge, London (2000)Google Scholar
  76. 76.
    Jenkins, H.: Convergence Culture: Where Old and New Media Collide. New York University, New York (2006)Google Scholar
  77. 77.
    Benkler, Y.: The Wealth of Networks: How Social Production Transform Markets and Freedom. Yale University Press, New Haven (2006)Google Scholar
  78. 78.
    Bourriaud, N.: Esthétique Relationnelle. Les Presses du Réel, Dijon (2002)Google Scholar
  79. 79.
    O’Neill, M., Hubbard, P.: Walking, sensing, belonging: ethno-mimesis as performative praxis. Vis. Stud. 25(1), 46–58 (2010)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Rensselaer Polytechnic InstituteTroyUSA

Personalised recommendations