Combinatorial Morphology in Visual Languages

Chapter
Part of the Studies in Morphology book series (SUMO, volume 4)

Abstract

Just as structured mappings between phonology and meaning make up the lexicons of spoken languages, structured mappings between graphics and meaning comprise lexical items in visual languages. Such representations may also involve combinatorial meanings that arise from affixing, substituting, or reduplicating bound and self-standing visual morphemes. For example, hearts may float above a head or substitute for eyes to show a person in love, or gears may spin above a head to convey that they are thinking. Here, we explore the ways that such combinatorial morphology operates in visual languages by focusing on the balance of intrinsic and distributional construction of meaning, the variation in semantic reference and productivity, and the empirical work investigating their cross-cultural variation, processing, and acquisition. Altogether, this work draws parallels between the visual and verbal domains that can hopefully inspire future work on visual languages within the linguistic sciences.

Keywords

Visual language Drawings Visual morphology Metaphor Combinatorial structure 

Notes

Acknowledgements

Kaitlin Pederson and Ryan Taylor are thanked for the brainstorming that contributed to this chapter, and Joost Schilperoord, Ray Jackendoff, and Charles Forceville are thanked for helpful feedback on earlier drafts.

References

  1. Abbott, M., and Ch. Forceville. 2011. Visual representation of emotion in manga: Loss of control is Loss of hands in Azumanga Daioh Volume 4. Language and Literature 20 (2): 91–112. https://doi.org/10.1177/0963947011402182.CrossRefGoogle Scholar
  2. Anderson, S.R. 1992. A-morphous morphology. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  3. Audring, J., and G. Booij. 2016. Cooperation and coercion. Linguistics 54 (4): 617–637.CrossRefGoogle Scholar
  4. Booij, G. 2010. Construction morphology. Oxford: Oxford University Press.Google Scholar
  5. Burr, D.C. 2000. Motion vision: Are “speed lines” used in human visual motion? Current Biology 10 (12): R440–R443. https://doi.org/10.1016/s0960-9822(00)00545-5. CrossRefGoogle Scholar
  6. Burr, D.C., and J. Ross. 2002. Direct evidence that “speedlines” influence motion mechanisms. The Journal of Neuroscience 22 (19): 8661–8664.CrossRefGoogle Scholar
  7. Carello, C., L.D. Rosenblum, and A. Grosofsky. 1986. Static depiction of movement. Perception 15 (1): 41–58.CrossRefGoogle Scholar
  8. Cohn, N. 2007. A visual lexicon. Public Journal of Semiotics 1 (1): 53–84.Google Scholar
  9. ———. 2010. Extra! Extra! Semantics in comics!: The conceptual structure of Chicago Tribune advertisements. Journal of Pragmatics 42 (11): 3138–3146.CrossRefGoogle Scholar
  10. ———. 2013a. Beyond speech balloons and thought bubbles: The integration of text and image. Semiotica 2013 (197): 35–63.CrossRefGoogle Scholar
  11. ———. 2013b. The visual language of comics: Introduction to the structure and cognition of sequential images. London: Bloomsbury.Google Scholar
  12. ———. 2016a. Linguistic relativity and conceptual permeability in visual narratives: New distinctions in the relationship between language(s) and thought. In The visual narrative reader, ed. N. Cohn, 315–340. London: Bloomsbury.Google Scholar
  13. ———. 2016b. A multimodal parallel architecture: A cognitive framework for multimodal interactions. Cognition 146: 304–323.CrossRefGoogle Scholar
  14. Cohn, N., and S. Ehly. 2016. The vocabulary of manga: Visual morphology in dialects of Japanese visual language. Journal of Pragmatics 92: 17–29.CrossRefGoogle Scholar
  15. Cohn, N., and S. Maher. 2015. The notion of the motion: The neurocognition of motion lines in visual narratives. Brain Research 1601: 73–84.CrossRefGoogle Scholar
  16. Cohn, N., B. Murthy, and T. Foulsham. 2016. Meaning above the head: Combinatorial constraints on the visual vocabulary of comics. Journal of Cognitive Psychology 28 (5): 559–574.CrossRefGoogle Scholar
  17. Culicover, P.W., and R. Jackendoff. 2012. Same-except: A domain-general cognitive relation and how language expresses it. Language 88 (2): 305–340.CrossRefGoogle Scholar
  18. Danesi, M. 2016. The semiotics of emoji: The rise of visual language in the age of the internet. London: Bloomsbury.Google Scholar
  19. Díaz Vera, J.E. 2013a. Embodied emotions in Medieval English language and visual arts. In Sensuous cognition explorations into human sentience: Imagination, (e)motion and perception, ed. R. Caballero and J.E. Diaz Vera, 195–219. Berlin: de Gruyter.Google Scholar
  20. ———. 2013b. Woven emotions: Visual representations of emotions in medieval English textiles. Review of Cognitive Linguistics 11 (2): 269–284.CrossRefGoogle Scholar
  21. Engelhardt, Y. 2002. The language of graphics. Dissertation, University of Amsterdam.Google Scholar
  22. Fauconnier, G., and M. Turner. 2002. The way we think: Conceptual blending and the mind’s hidden complexities. New York: Basic Books.Google Scholar
  23. Fay, N., M. Ellison, and S. Garrod. 2014. Iconicity: From sign to system in human communication and language. Pragmatics & Cognition 22 (2): 244–263.Google Scholar
  24. Forceville, Ch. 2005. Visual representations of the idealized cognitive model of anger in the Asterix album La Zizanie. Journal of Pragmatics 37 (1): 69–88.CrossRefGoogle Scholar
  25. ———. 2011. Pictorial runes in Tintin and the Picaros. Journal of Pragmatics 43 (3): 875–890.CrossRefGoogle Scholar
  26. ———. 2016. Conceptual metaphor theory, blending Tteory, and other cognitivist perspectives on comics. In The visual narrative reader, ed. N. Cohn, 89–114. London: Bloomsbury.Google Scholar
  27. Forceville, Ch., T. Veale, and K. Feyaerts. 2010. Balloonics: The visuals of balloons in comics. In The rise and reason of comics and graphic literature: Critical essays on the form, ed. J. Goggin and D. Hassler-Forest, 56–73. Jefferson: McFarland & Company, Inc.Google Scholar
  28. Forceville, Ch., E. El Refaie, and G. Meesters. 2014. Stylistics in comics. In The Routledge handbook of stylistics, ed. M. Burke, 485–499. London: Routledge.Google Scholar
  29. Friedman, S.L., and M.B. Stevenson. 1975. Developmental changes in the understanding of implied motion in two-dimensional pictures. Child Development 46: 773–778.CrossRefGoogle Scholar
  30. Garrod, S., N. Fay, J. Lee, J. Oberlander, and T. MacLeod. 2007. Foundations of representation: Where might graphical symbol systems come from? Cognitive Science 31 (6): 961–987.CrossRefGoogle Scholar
  31. Green, J. 2014. Drawn from the ground: Sound, sign and inscription in Central Australian sand stories. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  32. Gross, D., N. Soken, K.R. Rosengren, A.D. Pick, B.H. Pillow, and P. Melendez. 1991. Children’s understanding of action lines and the static representation of speed of locomotion. Child Development 62: 1124–1141.CrossRefGoogle Scholar
  33. Ito, H., T. Seno, and M. Yamanaka. 2010. Motion impressions enhanced by converging motion lines. Perception 39 (11): 1555–1561.CrossRefGoogle Scholar
  34. Jackendoff, R. 1990. Semantic structures. Cambridge, MA: MIT Press.Google Scholar
  35. ———. 2002. Foundations of language: Brain, meaning, grammar, evolution. Oxford: Oxford University Press.CrossRefGoogle Scholar
  36. Jackendoff, R., and J. Audring. 2016. Morphological schemas. The Mental Lexicon 11 (3): 467–493.CrossRefGoogle Scholar
  37. Katsuno, H., and C.R. Yano. 2002. Face to face: On-line subjectivity in contemporary Japan. Asian Studies Review 26 (2): 205–231.CrossRefGoogle Scholar
  38. Kawabe, T., and K. Miura. 2006. Representation of dynamic events triggered by motion lines and static human postures. Experimental Brain Research 175 (2): 372–375.CrossRefGoogle Scholar
  39. ———. 2008. New motion illusion caused by pictorial motion lines. Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) 55 (4): 228–234.CrossRefGoogle Scholar
  40. Kerr, S., and K. Durkin. 2004. Understanding of thought bubbles as mental representations in children with autism: Implications for theory of mind. Journal of Autism and Developmental Disorders 34 (6): 637–648.CrossRefGoogle Scholar
  41. Kim, H., and G. Francis. 1998. A computational and perceptual account of motion lines. Perception 27 (7): 785–797.CrossRefGoogle Scholar
  42. Koch, W.A. 1971. Varia semiotica. Hildensheim: Olms.Google Scholar
  43. Lakoff, G. 1992. The contemporary theory of metaphor. In Metaphor and thought, ed. A. Ortony, 202–251. Cambridge: Cambridge University Press.Google Scholar
  44. Lakoff, G., and M. Johnson. 1980. Metaphors we live by. Chicago: University of Chicago Press.Google Scholar
  45. McCloud, S. 1993. Understanding comics: The invisible art. New York: Harper Collins.Google Scholar
  46. Mori, K. 1995. The influence of action lines on pictorial movement perception in pre-school children. Japanese Psychological Research 27 (3): 183–187.CrossRefGoogle Scholar
  47. Munn, N.D. 1962. Walbiri graphic signs: An analysis. American Anthropologist 64 (5): 972–984.CrossRefGoogle Scholar
  48. ———. 1966. Visual categories: An approach to the study of representational systems. American Anthropologist 68 (4): 936–950.CrossRefGoogle Scholar
  49. ———. 1986/2016. The Warlbiri sand story. In The visual narrative reader, ed. N. Cohn, 230–249. London: Bloomsbury.Google Scholar
  50. Nakazawa, J. 2016. Manga literacy and manga comprehension in Japanese children. In The visual narrative reader, ed. N. Cohn, 157–184. London: Bloomsbury.Google Scholar
  51. Newton, D.P. 1985. Children’s perception of pictorial metaphor. Educational Psychology 5 (2): 179–185.CrossRefGoogle Scholar
  52. Ojha, A. 2013. An experimental study on visual metaphor. Dissertation, International Institute of Information Technology, Hyderabad, India.Google Scholar
  53. Parsons, S., and P. Mitchell. 1999. What children with autism understand about thoughts and thought bubbles. Autism 3 (1): 17–38.CrossRefGoogle Scholar
  54. Petersen, R.S. 2011. Comics, manga, and graphic novels: A history of graphic narratives. Santa Barbara: ABC-CLIO.Google Scholar
  55. Phillips, B.J., and E.F. McQuarrie. 2004. Beyond visual metaphor: A new typology of visual rhetoric in advertising. Marketing Theory 4 (1–2): 113–136.CrossRefGoogle Scholar
  56. Pratha, K., N. Avunjian, and N. Cohn. 2016. Pow, punch, pika, and chu: The structure of sound effects in genres of American comics and Japanese manga. Multimodal Communication 5 (2): 93–109.CrossRefGoogle Scholar
  57. Premack, D., and G. Woodruff. 1978. Does the chimpanzee have a ‘theory of mind’? Behavioral and Brain Science 4: 515–526.CrossRefGoogle Scholar
  58. Sadock, J.M. 1991. Autolexical syntax: A theory of parallel grammatical representations. Chicago: University of Chicago Press.Google Scholar
  59. Schilperoord, J. 2013. Raising the issue: A mental-space approach to Iwo Jima-inspired editorial cartoons. Metaphor and Symbol 28 (3): 185–212.CrossRefGoogle Scholar
  60. ———. 2017. Ways with pictures: Visual incongruities and metaphor. In Visual metaphor; structure and process, ed. G. Steen. Amsterdam: John Benjamins.Google Scholar
  61. Schodt, F.L. 1983. Manga! Manga! The world of Japanese comics. New York: Kodansha America Inc.Google Scholar
  62. Shinohara, K., and Y. Matsunaka. 2009. Pictorial metaphors of emotion in Japanese comics. In Multimodal metaphor, ed. Ch. Forceville and E. Uris-Aparisi, 265–293. Berlin/New York: Mouton De Gruyter.Google Scholar
  63. Stump, G.T. 2001. Inflectional morphology: A theory of paradigm structure. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  64. Talmy, L. 2000. Toward a cognitive semantics. Cambridge, MA: MIT Press.Google Scholar
  65. Walker, M. 1980. The lexicon of comicana. Port Chester: Comicana, Inc.Google Scholar
  66. Wellman, H.M., M. Hollander, and C.A. Schult. 1996. Young children’s understanding of thought bubbles and thoughts. Child Development 67: 768–788.CrossRefGoogle Scholar
  67. Wellman, H.M., S. Baron-Cohen, R. Caswell, J.C. Gomez, J. Swettenham, E. Toye, and K. Lagattuta. 2002. Thought-bubbles help children with autism acquire an alternative to a theory of mind. Autism 6 (4): 343–363.CrossRefGoogle Scholar
  68. Wichmann, S., and J. Nielsen. 2016. Sequential text-image pairing among the Classic Maya. In The visual narrative reader, ed. N. Cohn, 282–313. London: Bloomsbury.Google Scholar
  69. Wildfeuer, J., and J. Bateman. 2016. Linguistically oriented comics research in Germany. In The visual narrative reader, ed. N. Cohn, 19–66. London: Bloomsbury.Google Scholar
  70. Wilkins, D.P. 1997. Alternative representations of space: Arrernte narratives in sand. In Proceedings of the CLS opening academic year ’97 ’98, ed. M. Biemans and J. van de Weijer, 133–164. Nijmegen: Center for Language Studies.Google Scholar
  71. ———. 1997/2016. Alternative representations of space: Arrernte narratives in sand. In The visual narrative reader, ed. N. Cohn, 252–281. London: Bloomsbury.Google Scholar
  72. Yannicopoulou, A. 2004. Visual aspects of written texts: Preschoolers view comics. L1- Educational Studies in Language and Literature 4: 169–181.CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Communication and CognitionTilburg UniversityTilburgThe Netherlands

Personalised recommendations