Skip to main content
SpringerLink
Log in

Augmented Narrative: Assisting the Reader with Sound

  • Chapter
  • First Online:
Assistive Augmentation

Part of the book series: Cognitive Science and Technology ((CSAT))

  • 877 Accesses

Abstract

Augmented Narrative is literature that assists readers by augmenting the narrative text with sound effects. The assistance comes when the reader is not being engaged into the storyline, using smart glasses as a biofeedback that measure levels of engagement through the temperature of the nose. The concept emerges from literary theory, translated into cognitive and computational terms, and put into a framework of cognitive ecology to design literature that provides with an embodied experience of the storyworld.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 129.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Angelotti M, Behnke RR, Carlile LW (1975) Heart rate: a measure of reading involvement. Res Teach Engl 192–199

    Google Scholar 

  2. Ashby J, Rayner K (2012) Reading in alphabetic writing systems: evidence from cognitive neuroscience. Neurosci Educ Good Bad Ugly 61

    Google Scholar 

  3. Benton M (2005) Reader-response criticism. Understanding children’s literature: key essays from the second edition of the international companion encyclopedia of children’s literature, 86

    Google Scholar 

  4. Berka C, Levendowski DJ, Lumicao MN, Yau A, Davis G, Zivkovic VT, Olmstead RE, Tremoulet PD, Craven PL (2007) Eeg correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviat Space Environ Med 78(Supplement 1):B231–B244

    Google Scholar 

  5. Bertelson P, de Gelder B (2004) The psychology of multimodal perception. Crossmodal space and crossmodal attention, pp 141–177

    Google Scholar 

  6. Biedert R, Buscher G, Schwarz S, Möller M, Dengel A, Lottermann T (2010) The text 2.0 framework. In: Workshop on eye gaze in intelligent human machine interaction. Citeseer, pp 114–117

    Google Scholar 

  7. Cannon WB (1927) The james-lange theory of emotions: A critical examination and an alternative theory. Am J Psychol 106–124

    Google Scholar 

  8. Cohen B, Izard C, Facial Simons R (1986) physiological indexes of emotions in mother-infant interactions. In Psychophysiology, vol. 23, Soc Psychophysiol Res, (1010) Vermont Ave Nw Suite 1100. Washington, DC 20005:429–429

    Google Scholar 

  9. Corradini A, Antonietti A (2013) Mirror neurons and their function in cognitively understood empathy. Conscious Cogn 22(3):1152–1161

    Article  Google Scholar 

  10. Coulson S, King JW, Kutas M (1998) Expect the unexpected: event-related brain response to morphosyntactic violations. Lang Cogn Process 13(1):21–58

    Article  Google Scholar 

  11. Csikszentmihalyi M, Csikzentmihaly M (1991) Flow: the psychology of optimal experience, vol 41. Harper Perennial, New York

    Google Scholar 

  12. Eason SH, Goldberg LF, Young KM, Geist MC, Cutting LE (2012) Reader-text interactions: How differential text and question types influence cognitive skills needed for reading comprehension. J Educ Psychol 104(3):515

    Article  Google Scholar 

  13. Ekman P, Levenson RW, Friesen WV (1983) Autonomic nervous system activity distinguishes among emotions. Sci 221(4616):1208–1210

    Article  Google Scholar 

  14. Engle RW, Tuholski SW, Laughlin JE, Conway AR (1999) Working memory, short-term memory, and general fluid intelligence: a latent-variable approach. J Exp Psychol Gen 128(3):309

    Article  Google Scholar 

  15. Ericsson KA, Kintsch W (1995) Long-term working memory. Psychol Rev 102(2):211

    Article  Google Scholar 

  16. Fludernik M (2002) Towards a ‘natural’ narratology. Routledge

    Google Scholar 

  17. Folkerth W (2014) The sound of Shakespeare. Routledge

    Google Scholar 

  18. Gibson JJ. The senses considered as perceptual systems

    Google Scholar 

  19. Gibson JJ (2014) The ecological approach to visual perception: classic edition. Psychology Press

    Google Scholar 

  20. Gladwell M (2007) Blink: the power of thinking without thinking. Back Bay Books

    Google Scholar 

  21. Goldman AI (2005) 2 imitation, mind reading, and simulation. Perspectives on imitation: imitation, human development, and culture, 79

    Google Scholar 

  22. Gouvea AC. How to examine the p600 using language theory: what are the syntactic processes reflected in this component?

    Google Scholar 

  23. Graesser AC, McNamara DS (2011) Computational analyses of multilevel discourse comprehension. Top Cogn Sci 3(2):371–398

    Article  Google Scholar 

  24. Havelock EA (1980) The oral composition of greek drama. Quaderni Urbinati di Cultura Classica, 61–113

    Google Scholar 

  25. Healy SD (1984) Boredom, self, and culture. Fairleigh Dickinson University Press

    Google Scholar 

  26. Herman D (2000) Narratology as a cognitive science. Image Narrative 1:1

    MathSciNet  Google Scholar 

  27. Hollan J, Hutchins E, Kirsh D (2000) Distributed cognition: toward a new foundation for human-computer interaction research. ACM Trans Comput Hum Interact (TOCHI) 7(2):174–196

    Google Scholar 

  28. Houix O, Lemaitre G, Misdariis N, Susini P, Urdapilleta I (2012) A lexical analysis of environmental sound categories. J Exp Psychol Appl 18(1):52

    Article  Google Scholar 

  29. Hutchins E (2010) Cognitive ecology. Topics. Cogn Sci 2(4):705–715

    Article  Google Scholar 

  30. Ioannou S, Ebisch S, Aureli T, Bafunno D, Ioannides HA, Cardone D, Manini B, Romani GL, Gallese V, Merla A. The autonomic signature of guilt in children: a thermal infrared imaging study

    Google Scholar 

  31. Ioannou S, Gallese V, Merla A (2014) Thermal infrared imaging in psychophysiology: potentialities and limits. Psychophysiology 51(10):951–963

    Article  Google Scholar 

  32. Jalilehvand M (2012) The effects of text length and picture on reading comprehension of iranian efl students. Asian Soc Sci 8(3):329

    Article  Google Scholar 

  33. Jennett C, Cox AL, Cairns P, Dhoparee S, Epps A, Tijs T, Walton A (2008) Measuring and defining the experience of immersion in games. Int J Hum Comput Stud 66(9):641–661

    Article  Google Scholar 

  34. Kadir D (2006) Comparative literature in a world become tlon. Comp Crit Stud 3(1):125–138

    Article  Google Scholar 

  35. Keller J, Bless H, Blomann F, Kleinböhl D (2011) Physiological aspects of flow experiences: skills-demand-compatibility effects on heart rate variability and salivary cortisol. J Exp Soc Psychol 47(4):849–852

    Article  Google Scholar 

  36. Kiesling LL (2012) Mirror neuron research and adam smith’s concept of sympathy: three points of correspondence. Rev Austrian Econ 25(4):299–313

    Article  Google Scholar 

  37. Kintsch W (1988) The role of knowledge in discourse comprehension: a construction-integration model. Psychol Rev 95(2):163

    Article  Google Scholar 

  38. Kintsch W, Patel VL, Ericsson KA (1999) The role of long-term working memory in text comprehension. Psychologia 42(4):186–198

    Google Scholar 

  39. Kunze K, Sanchez S, Dingler T, Augereau O, Kise K, Inami M, Tsutomu T (2015) The augmented narrative: toward estimating reader engagement. In: Proceedings of the 6th augmented human international conference. ACM, pp 163–164

    Google Scholar 

  40. Lebrecht J, Kaye D (1999) The art and technique of design, sound and music for the theatre

    Google Scholar 

  41. Lemaitre G, Heller LM (2013) Evidence for a basic level in a taxonomy of everyday action sounds. Exp Brain Res 226(2):253–264

    Article  Google Scholar 

  42. Lewis D (1978) Truth in fiction. Am Philos Q 15(1):37–46

    Google Scholar 

  43. MacDonald RA, Hargreaves DJ, Miell D, Davidson JW, North AC (2002) Musical identities, vol 13. Oxford University Press, Oxford

    Google Scholar 

  44. MacLuhan M (2005) The effect of the printed book on language in the 16th century. Gingko Press

    Google Scholar 

  45. Mar RA, Oatley K (2008) The function of fiction is the abstraction and simulation of social experience. Perspect Psychol Sci 3(3):173–192

    Article  Google Scholar 

  46. McAdams SE, Bigand EE (1993) Thinking in sound: the cognitive psychology of human audition. In: Based on the fourth workshop in the tutorial workshop series organized by the hearing group of the French acoustical society. Clarendon Press/Oxford University Press

    Google Scholar 

  47. McCraty R, Atkinson M, Tomasino D, Bradley RT (2009) The coherent heart: heart-brain interactions, psychophysiological coherence, and the emergence of system-wide order. Integr Rev 5(2):10–115

    Google Scholar 

  48. Mcluhan HM (2010) Understanding me: lectures and interviews. McClelland & Stewart

    Google Scholar 

  49. Merleau-Ponty M, Smith C (1996) Phenomenology of perception. Motilal Banarsidass Publishe

    Google Scholar 

  50. Michaels CF, Carello C (1981) Direct perception. Prentice-Hall Englewood Cliffs, NJ

    Google Scholar 

  51. Moreno R, Mayer RE (2002) Verbal redundancy in multimedia learning: when reading helps listening. J Educ Psychol 94(1):156

    Article  Google Scholar 

  52. Nakanishi R, Imai-Matsumura K (2008) Facial skin temperature decreases in infants with joyful expression. Infant Behav Dev 31(1):137–144

    Article  Google Scholar 

  53. Or CK, Duffy VG (2007) Development of a facial skin temperature-based methodology for non-intrusive mental workload measurement. Occup Ergon 7(2):83

    Google Scholar 

  54. Paivio A (1978) A dual coding approach to perception and cognition. Modes of perceiving and processing information, pp 39–51

    Google Scholar 

  55. Pauletto S (2014) Film and theatre-based approaches for sonic interaction design. Digit Creativity 25(1):15–26

    Article  Google Scholar 

  56. Penney CG (1989) Modality effects and the structure of short-term verbal memory. Mem Cogn 17(4):398–422

    Article  Google Scholar 

  57. Pollard AW (2010) Shakespeare’s fight with the pirates and the problems of the transmission of his text. Cambridge University Press

    Google Scholar 

  58. Purves AC, Rippere V. Elements of writing about a literary work—a study of response to literature

    Google Scholar 

  59. Rabinowitz PJ (1997) Before reading: narrative conventions and the politics of interpretation. The theory and interpretation of narrative series, ERIC

    Google Scholar 

  60. Rose PA (2012) The classical trivium: the place of thomas nashe in the learning of his time (by marshall mcluhan). Can J Commun 37:4

    Article  Google Scholar 

  61. Ryan M-L (1991) Possible worlds, artificial intelligence, and narrative theory. Indiana University Press

    Google Scholar 

  62. Sadoski M (1983) An exploratory study of the relationships between reported imagery and the comprehension and recall of a story. Read Res Q 110–123

    Google Scholar 

  63. Schütz A (1951) Making music together: a study in social relationship. Soc Res 76–97

    Google Scholar 

  64. Scolari CA (2012) Media ecology: exploring the metaphor to expand the theory. Commun Theory 22(2):204–225

    Article  Google Scholar 

  65. Shokoff J (2001) What is an audiobook? J Popular Cult 34(4):171

    Article  Google Scholar 

  66. Sowa JF. Conceptual structures: information processing in mind and machine

    Google Scholar 

  67. Spacks PM (1995) Boredom: the literary history of a state of mind. University of Chicago Press

    Google Scholar 

  68. Thierry G, Martin CD, Gonzalez-Diaz V, Rezaie R, Roberts N, Davis PM (2008) Event-related potential characterisation of the shakespearean functional shift in narrative sentence structure. Neuroimage 40(2):923–931

    Article  Google Scholar 

  69. Tribble E, Sutton J et al (2011) Cognitive ecology as a framework for shakespearean studies. Shakespeare Stud 39:94–103

    Google Scholar 

  70. Verstraete P. The frequency of imagination: auditory distress and aurality in contemporary music theatre

    Google Scholar 

  71. Walter O (1982) Orality and literacy. TJPress, London, The technologizing of the word

    Google Scholar 

  72. Weaver P (2012) Radio drama: a “visual sound” analysis of John, George and drew baby. PhD thesis, University of Central Florida Orlando, Florida

    Google Scholar 

  73. Zajonc RB (1985) Emotion and facial efference: a theory reclaimed. Science 228(4695):15–21

    Article  Google Scholar 

  74. Zhao H (2012) Emotion-driven interactive storytelling. PhD thesis, Bournemouth University

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. KMD, Keio University, Tokyo, Japan

    Susana Sanchez Perez, Naohito Okude & Kai Kunze

Authors
  1. Susana Sanchez Perez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Naohito Okude
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kai Kunze
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Susana Sanchez Perez , Naohito Okude or Kai Kunze .

Editor information

Editors and Affiliations

  1. Synaptics, Zug, Switzerland

    Jochen Huber

  2. Department of Computer Science, Stony Brook University, New York, New York, USA

    Roy Shilkrot

  3. Massachusetts Institute of Technology (MIT), Cambridge, Massachusetts, USA

    Pattie Maes

  4. Augmented Human Lab, Singapore University of Technology and Design, Singapore, Singapore

    Suranga Nanayakkara

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Sanchez Perez, S., Okude, N., Kunze, K. (2018). Augmented Narrative: Assisting the Reader with Sound. In: Huber, J., Shilkrot, R., Maes, P., Nanayakkara, S. (eds) Assistive Augmentation. Cognitive Science and Technology. Springer, Singapore. https://doi.org/10.1007/978-981-10-6404-3_5

Download citation

  • DOI: https://doi.org/10.1007/978-981-10-6404-3_5

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-10-6402-9

  • Online ISBN: 978-981-10-6404-3

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation