Advertisement

Media Poetics and Cognition in Colocative Audiovisual Displays

  • Michael FilimowiczEmail author
Chapter
Part of the Human–Computer Interaction Series book series (HCIS)

Abstract

Colocative display is the technique of sonically articulating the screen area of audiovisual media by dynamically placing and animating the associated sounds in spatial localization to their visual cues. Two such systems have been designed to date, the author’s Pixelphonics system and the Allosphere facility at University of California at Santa Barbara. While both technologies are prototypes, and thus lacking in a rich historical tradition that might inform what film theorist David Bordwell has called historical poetics, Bordwell’s concept of analytical and theoretical poetics can be fruitfully brought to bear to elucidate the general principles for making colocative audiovisual media and applications. The poetics, or ‘principles of making’ colocative media are situated within a discussion of the empirical dimensions of auditory localization, cognition and attentional resources, general audiovisual practices, acoustics and phenomenology. A new concept, that of the soundscene, is introduced to hone in on the particular design affordances of colocative displays. This inquiry blends second and third wave HCI approaches in its hybridization of humanist media theories with cognitivist-attentional usability perspectives.

References

  1. Allosphere Research Facility (2017) University of California Santa Barbara. http://www.allosphere.ucsb.edu/index.html
  2. Altman R (1992) Sound space. In: Altman R (ed) Sound theory sound practice. Routledge, New York/LondonGoogle Scholar
  3. Belton J (1992) 1950s magnetic sound: the frozen revolution. In: Altman R (ed) Sound theory sound practice. Routledge, New York/LondonGoogle Scholar
  4. Blindsight. Merriam-Webster Medical Dictionary Online: https://www.merriam-webster.com/medical/blindsight
  5. Bordwell D (2002) Intensified continuity visual style in contemporary American film. Film Q 55(3):16–28CrossRefGoogle Scholar
  6. Bordwell D (2007) The poetics of cinema. Routledge, New YorkGoogle Scholar
  7. Bregman A (1990) Auditory scene analysis. MIT Press, Cambridge, MAGoogle Scholar
  8. Bresson R (1985) Notes on sound. In: Weis E, Belton J (eds) Film sound: theory and practice. Columbia University Press, New YorkGoogle Scholar
  9. Carlile S, Leung J (2016) The perception of auditory motion. Trends Hearing 20:1–19CrossRefGoogle Scholar
  10. Chion M (2013) Sensory aspects of contemporary cinema. In: Richardson J, Gorbman C, Vernallis C (eds) The Oxford handbook of new audiovisual aesthetics. Oxford University Press, Oxford. Amazon Kindle editionGoogle Scholar
  11. Coleman S (2005) Badgered. On: a collection of 2005 academy award nominated short films (DVD). Magnolia Home EntertainmentGoogle Scholar
  12. Connor S (2013) Sounding out film. In: Richardson J, Gorbman C, Vernallis C (eds) The Oxford handbook of new audiovisual aesthetics. Oxford University Press, Oxford. Amazon Kindle editionGoogle Scholar
  13. Donnelly KJ (2013) Extending film aesthetics: audio beyond visuals. In: Richardson J, Gorbman C, Vernallis C (eds) The Oxford handbook of new audiovisual aesthetics. Oxford University Press, Oxford. Amazon Kindle editionGoogle Scholar
  14. Gibson JJ (1966) The senses considered as perceptual systems. Houghton Mifflin Co., BostonGoogle Scholar
  15. Godfroy M, Roumes C, Dauchy P (2003) Spatial variations of visual-auditory fusion areas. Perception 32:1233–1245CrossRefGoogle Scholar
  16. He JJ (2017) Literature review on spatial audio. In: Gan WS, Kuo CCJ, Zheng TF, Barni M (eds) Spatial audio reproduction with primary ambient extraction, Springer briefs in signal processing. Springer, BaselCrossRefGoogle Scholar
  17. Herbert R (2017) Sonic subjectivities. In: Cobussen M, Meelberg V, Truax B (eds) The Routledge companion to sounding art. Routledge, New York. Amazon Kindle editionGoogle Scholar
  18. Höllerer T, Kuchera-Morin J, Amatriain X (2007) The allosphere: a large-scale immersive surround-view instrument. In: EDT’07: proceedings of the 2007 workshop on emerging displays technologies: images and beyond: the future of displays and interaction, San Diego, CA, USA, Aug, 2007, Article No. 3. doi 101145/12782401278243Google Scholar
  19. Holman T (2007) Surround sound up and running. Focal Press, New YorkGoogle Scholar
  20. Horst S (2005) Phenomenology and psychophysics. Phenomenol Cogn Sci 4:1–21CrossRefGoogle Scholar
  21. Kerins M (2010) Beyond Dolby (stereo): cinema in the digital sound age. Indiana University Press, BloomingtonGoogle Scholar
  22. Kim-Cohn S (2017) Dams, weirs, and damn weird ears– post-ergonal sound. Auditory capital, media publics and the sounding arts. In: Cobussen M, Meelberg V, Truax B (eds) The Routledge companion to sounding art. Routledge, New York. Amazon Kindle editionGoogle Scholar
  23. Kuchera-Morin J, Wright M, Wakefield G, Roberts C, Adderton D, Sajadi B, Höllerer T, Majumder A (2014) Immersive full-surround multi-user system design. Comput Graph 40:10–21CrossRefGoogle Scholar
  24. LaBelle B (2017) Restless acoustics, emergent publics. In: Cobussen M, Meelberg V, Truax B (eds) The Routledge companion to sounding art. Routledge, New York. Amazon Kindle editionGoogle Scholar
  25. Lacey K (2017) Auditory capital, media publics and the sounding arts. In: Cobussen M, Meelberg V, Truax B (eds) The Routledge companion to sounding art. Routledge, New York. Amazon Kindle editionGoogle Scholar
  26. Makous JC, Middlebrooks JC (1990) Two-dimensional sound localization by human listeners. J Acoust Soc Am 87:2188–2200CrossRefGoogle Scholar
  27. McGovern DP, Roudaia E, Newell FN, Roach NW (2016) Perceptual learning shapes multisensory causal inference via two distinct mechanisms. Sci Rep 6 Article number 24673. https://www.nature.com/articles/srep24673
  28. Middlebrooks JC, Green DM (1991) Sound localization by human listeners. Annu Rev Psychol 42:135–159CrossRefGoogle Scholar
  29. Richardson DC, Spivey MJ (2000) Representation, space and Hollywood Squares: looking at things that aren’t there anymore. Cognition 76(3):269–295CrossRefGoogle Scholar
  30. Stillwell RJ (2013) Audiovisual space in an era of technological convergence. In: Richardson J, Gorbman C, Vernallis C (eds) The Oxford handbook of new audiovisual aesthetics. Oxford University Press, Oxford. Amazon Kindle editionGoogle Scholar
  31. Tappeiner H (2013) Robotics: what is the difference between tactile feedback and haptic feedback? Quora.com. https://www.quora.com/Robotics-What-is-the-difference-between-tactile-feedback-and-haptic-feedback
  32. Ursino M, Magosso E, Cuppini C (2011) Sensory fusion. In: Cutsuridis V, Hussain A, Taylor JG (eds) Perception-action cycle: models, architectures, and hardware. Springer series in cognitive and neural systems. Springer International Publishing AG, Basel, pp 23–62CrossRefGoogle Scholar
  33. Wickens CD (2002) Multiple resources and performance prediction. Theoretical issues in ergonomic. Science 3(2):166Google Scholar
  34. Wickens CD (2008) Multiple resources and mental workload. Hum Factors 50(3):449–455CrossRefGoogle Scholar
  35. Wright RD, Ward LM (2008) Orienting of attention. Oxford University Press, OxfordGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of Interactive Arts & TechnologySimon Fraser UniversitySurreyCanada

Personalised recommendations