Keywords

Everyday communication is full of simple and sometimes also complex narratives that make our lives and our surroundings comprehensible. These narratives are realized in a large variety of media types. On a typical day, I may meet one of my neighbors on my way to the mailbox and she will tell me, using shifting tones and intonations and sometimes vivid gestures, about a weekend trip to her relatives in Gothenburg. At the breakfast table, I read in the newspaper about all sorts of events: accidents and happenings in my region and political developments around the world. The written texts are visually formed in different ways and often combined or integrated with still images that contribute to or tell their own stories. Listening to the radio while brushing my teeth and feeding my cats, I first hear a song about a problematic love affair and then an instrumental piece that depicts tensions among several emotional states. When I finally reach my worktable and start my computer, I receive an email from my daughter with a link to a short movie that I decide to watch before starting to work properly. The email tells a short story about the movie, which represents the adventures of a computer mouse.

Therefore, long before I have even had my first cup of coffee I have perceived an abundance of narratives. It is clear that many of these narratives are, or may be, connected to each other. They do not exist in isolation, and there are clearly no definite borders between narratives formed by dissimilar media types. Considering that there are no definitive borders between media types as such—they all overlap each other, in complex patterns of similarities and differences—this is hardly surprising. Media obviously have their communicative capacities because of our cognitive faculties, and it is almost absurd to suggest the notion of a cognitive system working in such a way that representations of events through one kind of medium could not in any way be matched by representations of events through other media forms. A brain that harbors a cognitive system composed of secluded, media-specific strata of information would be dysfunctional.

However, we do have the capacity to communicate about things through different forms of media in such a way that narratives in various media types connect to each other in highly meaningful ways. These connections may be immediate, such as when my neighbor’s speech and gestures together narrate about a trip, or when the written texts and still images in the newspaper narrate about a political crisis. However, the connections may also cover temporal gaps: my daughter’s email may include a description of the movie that makes it possible for me to anticipate what goes on in it, and it is also possible for me to later tell my wife about what happened in the piece of music that I listened to in the morning.

Basic Terminology

Several complex capacities and phenomena are involved in these communicative occurrences. At the heart of the matter is how narratives may be created beyond specific media types or be moved across media borders; therefore, it is convenient to use terms starting with the Latin prefix ‘trans-’—which means ‘beyond’, ‘across’, or ‘through’—to denote what is going on. As there are several expressions containing ‘trans-’ and ‘media’, I will briefly comment on some central terms here and tie them to succinct conceptual definitions.

From the most wide-ranging perspective, the term ‘transmediality’ should be understood as referring to the general concept that different media types share many basic traits that can be described in terms of material properties and abilities for activating mental capacities. All media products, in partly similar ways, are physical existences that trigger semiotic activity and can be properly understood only in relation to each other. Thus, physical media properties and semiosis are transmedial phenomena. More specifically, different media types may, to a large extent—although certainly not completely—communicate similar things, such as events forming narratives. Using more technical language, several media types may more or less fully represent “compound media characteristics” of various sorts (Elleström 2014a: 39–45). In other words, represented media characteristics may be transmedial to different degrees. Transmediality is evidently a central part of intermediality, which is an even broader concept based on the proposition that different media types are interrelated in all kinds of ways.

It is only a short step from the idea that represented media characteristics may be transmedial to different degrees to recognizing that media characteristics, because of their transmedial nature, can be understood as being transferred among different kinds of media. Inserting a temporal perspective, it very often makes sense to acknowledge not only that similar media characteristics are or may be represented by dissimilar media but also that media characteristics that can in some respect be understood as the same, recur after having appeared in another medium. The examples of a written email describing the events in a movie and spoken words retelling a musical story both include a temporal gap between what might be called source and target media, but also the implicit notion of sameness. We find the relations between email and movie and between speech and music meaningful because the events that they represent are not only similar but in some respects the same (here, sameness should be understood as a pragmatic rather than a metaphysical quality). I refer to such a transfer of media characteristics as transmediation. In our minds, some perceived media characteristics of the target medium are, in important ways, the same as those of the source medium, which is to say that the media characteristics of an initial medium are perceived to be represented again by another kind of medium (Elleström 2014a: 20–27).

Building on these brief stipulations, the term ‘transmedial narration’ should be understood to refer to all varieties of transmediality and transmediation where narration is a media characteristic that is significant enough to be observed. In the most general terms, then, the concept of transmedial narration includes the notion that an abundance of different media types share traits that give them narrative capacities. In more specific terms, transmedial narration also includes the idea that the world is actually full of various sorts of more or less developed and complex narratives communicated by different media types. In its most particular sense, transmedial narration can be understood as transmediation of narratives; the characteristics of narratives can be represented again by dissimilar media types and yet be perceived to be the same despite the transfer.

Transmedial narration, in its most general sense, must be accepted as a reality that has a bearing on a lot of communication. Furthermore, transmediation of narratives is extremely common, not only in everyday communication but also in more complex and official systems of communication such as education, research, and legal processes. It also flourishes in religion, art, and entertainment.

For some years now, Henry Jenkins’s concept of transmedia storytelling has been popular. This concept refers to the modern phenomenon of building large narratives as a sum of partial narratives distributed by different kinds of media such as motion pictures, comics, video games, novels, and various forms of Internet-based media: “A transmedia story unfolds across multiple media platforms with each new text making a distinctive and valuable contribution to the whole” (Jenkins 2008: 97–98). In fact, this is an old and widespread phenomenon that can be observed in, for instance, Hindu, Greek, and Christian mythologies, although historical and cultural differences can obviously be noted (see for instance Ryan 2013; Mittell 2014). Transmedia storytelling—narratives in different media types working together to form a larger whole—requires that narratives can be largely transmediated. It would not be possible to combine narratives from different media types to a larger whole if these narratives did not overlap. In effect, this means that one recognizes represented media characteristics in the different media as the same; thus, represented persons, environments, ideas, events, and their interrelations can interlock. However, current research in transmedia storytelling does not engage what I consider to be the central questions of transmedial narration: how are such transmediations possible at all and what are their limitations?

Earlier Research

Although views differ considerably regarding most aspects of transmedial narration, the majority of researchers within the area seem to agree that narration is indeed a transmedial concept: media types that are not language-based may also narrate to a certain extent. Narration in various non-verbal media types has been noted, discussed, and to a certain degree theorized for centuries, although it has not been thoroughly conceptualized until recently. Explicit recognition of narration as a transmedial phenomenon can be found from the 1960s onward (Bremond 1964; Barthes 1977 [1966]).

As media characteristics are clearly not either fully transmedial or not transmedial at all, the extent to which and ways in which narration is transmedial has been a principal question since the advent of thorough explorations of the concept. Seymour Chatman, often quoted in the research on transmedial narration, maintained that the “transposability of the story is the strongest reason for arguing that narratives are indeed structures independent of any medium” (Chatman 1978: 20; cf. Altman 2008: 1). Several commentators consider this to be an overstatement; because dissimilar media types have different means for communicating narratives, narrative structures cannot be understood as ‘independent’ of medium in a strong sense: “Narratives are not so much structures independent of any medium, as structures common to several media” (Walsh 2007: 63). Thus, narratives always depend on some kind of medium to be realized; however, as different media types may generate narratives that are nevertheless recognizable as the same, narrative structures can be understood to be ‘independent’ of the medium, in a weak sense. Although media differences certainly do make a difference, the “transposability of the story” (Chatman 1978: 20) remains.

To explain the relative dependency on media types, it is imperative to realize that the partly dissimilar and partly shared properties of various media types “both open up possibilities and impose constraints [that] shape the narration” (Rimmon-Kenan 1989: 160)—even though narration is “a process which is not in its basic aims specific to any medium [it] deploys the materials and procedures of each medium for its ends” (Bordwell 1985: 49). Marie-Laure Ryan, who has played an important role in the development of what she calls transmedial narratology, has articulated a nuanced view on the relation between media types and narration: “A core of meaning may travel across media, but its narrative potential will be filled out, actualized differently when it reaches a new medium” (Ryan 2005: 1). Although I share the general views of Rimmon-Kenan, Bordwell, and Ryan on this matter, my way of conceptualizing transmedial narration in this treatise will differ substantially from all of the scholars quoted above (as will be demonstrated in Chaps. 2, 3, and 4).

The questions of how and to what extent narration is transmedial have also been debated in a more implicit way, through a steady flow of articles and books on narration in ever-new media types. Previous investigation in the area of transmedial narration raises some hope that research will be able to map and integrate knowledge about a very large range of different narrative media types. The following list of examples of media that have been investigated from a narratological perspective since the 1970s (especially highlighting early studies of the respective media types) demonstrates impressive breadth. Apart from numerous studies of narration in various forms of written literature, there have been studies on narration in spoken, everyday language (Labov 1972); comics (Hünig 1974; Abbott 1986); painting (Alpers 1976; Steiner 1988: 7–42; Wolf 2003, 2004); literature and film (Chatman 1978); comics and image sequences (Schnackertz 1980); written language, painting, and film (Goodman 1981); history writing (White 1981); painting and reliefs on ancient urns, walls, columns, and sarcophagi (Brilliant 1984); film music (Gorbman 1987); instrumental music (Newcomb 1987; Kramer 1991); drama (Richardson 1988); television news (Campbell and Reeves 1989); written and oral language in the legal system (Brooks and Gewirtz 1998); literature and history writing (Canary and Kozicki 1978; Cohn 1990); mural decoration in churches (Lavin 1990); written language in economics and the natural sciences (Nash 1990); music in general and opera (Abbate 1991; McClatchie 1997); advertisements (Stern 1994); spoken, everyday language and literature (Fludernik 1996); dance (Foster 1996); painting and photography (Kafalenos 1996); maps, diagrams, and advertisements (Kress and van Leeuwen 1996: 45–78); family photographs (Hirsch 1997); hypertext (Hayles 2001); still images and moving images (Ribière and Baetens 2001); film and television programs (Thompson 2003); all kinds of artistic media (Gaudreault and Marion 2004); computer games (Neitzel 2005); radio broadcasts of sports events (Ryan 2006); literature, comics, film, radio play, and hypertext (Mahne 2007); visual diagrams (Ryan 2007); music, literature, and drama (Almén 2008); written and oral language about the self (Eakin 2008); architecture and literature (Psarra 2009); sculpture (Wolf 2011); mathematical proofs (Doxiadis 2012); mathematical diagrams (de Freitas 2012); novels including photographs (Schwanecke 2012); journalism (Berning 2014); music and dance (Kutschke 2015); program music (Liu 2015); musical concept albums and their sleeves (Arvidson 2016); music combined with moving images (Giannoukakis 2016); and board games (Thibault 2016).

This breadth is emphasized even more if one considers the many collections of articles with really wide-ranging scopes of narrative media types. However, as Jan-Noël Thon accurately noted, “a genuinely transmedial narratology is not (or should not be) the same as a collection of media-specific narratological terms and concepts” (Thon 2016: 15). With some exceptions, this succinct observation can be used to criticize the setup of edited collections on narration in visual art, music, poetry, film, and comics (Nünning and Nünning 2002); face-to-face communication, gestures, painting, comics, moving pictures, music, and digital media (Ryan 2004b); literature, comics, television serials, motion pictures, photonovels, photo collections, interactive writing on the Internet, computer games, and advertisements in television and radio (Grishakova and Ryan 2010); computer-mediated communication, video games, political speeches, and film (Hoffmann 2010); television programs, motion pictures, web texts, opera, comics, speech, gesture, and multimodal novels (Page 2010); drama, feature films, graphic novels, video games, literature, visual art, and television series (Ryan and Thon 2014); literature, drama, film, journalism, television news, law courts, and oral communication (Nünning 2015); and film music, songs, rock music, radio drama, video games, audio books, and audio guides (Mildorf and Kinzel 2016b).

Despite this remarkable range, most research on narration is concentrated on verbal media types, with robust narratological traditions in both linguistics and the study of literature. Furthermore, artistic media types, whether they include verbal components or not, dominate the research field. More importantly, one must conclude that the various perspectives on a wealth of media types are anything but broadly integrated. As Thon noted (2015: 441), monographs and articles on transmedial narration, as intermedial research at large—with only a few exceptions (such as Ryan 2006)—consider just one or two media types at a time and are thus transmedial only in a weak sense.

While transmedial narratology is widespread to the extent that there are many studies of narration outside the verbal domain, the field is largely unexplored considering the few studies that seek to discover the common ground of narration in a broad range of media types. Although there has been a lot of research for several decades on transmedial narration, there is presently only a piecemeal understanding of how the fundamentals of a genuinely transmedial theory of narration might look. This is remarkable considering the ubiquity of narratives in various media types in general, and more specifically transmediated narratives in all forms of communication. Notwithstanding several decades of research on the fundamental importance of narratives for humans, a truly transmedial conceptualization of the central features of narration is still lacking. There are still no large-scale studies on transmedial narration in a strong sense, in which truly transmedial concepts useful for analyzing narration in a really broad scope of media types are methodically developed. Considering the diverging conceptualizations in the many studies on narration in various media, I conclude that such a fundamental conceptual framework is much needed.

This is not to say that narration is necessarily a completely transmedial phenomenon. Liviu Lutas’s contention that “narration in general can [not] be considered to be a trait that can be found in all media” (Lutas 2016: 33) should be taken seriously. Clearly, not all kinds of media can narrate to the same degree; consequently, there are probably media types whose narrative capacities are so rudimentary that they might just as well be considered non-narrative. For the moment, however, I leave the possible borders of transmediality in narration open so that I can explore the whole media territory without restrictions.

Aims and Goals

It became clear in the previous section that the questions of how and to what extent narration is transmedial have been central in earlier research. They will remain so in this study. I have also already mentioned a couple of related questions that I find imperative: How are transmediations of narratives possible at all, and what are their limitations? The treatise will also be guided by some more specific research questions: How can narration be conceptualized within a broad communicative context, including psychological and cognitive aspects? How can narration be conceptualized so that both its wide-ranging transmedial potential and its media-specific limitations can be accounted for? What are the main constituents of narration understood in a truly transmedial way and how are they related to other central features of communication?

An eloquent quotation from Liv Hausken, expressing views that overlap closely with my own, shows the direction of my aim with this treatise:

I believe that we should aspire to narrative theories that are independent of medium, while recognizing that the development of such theories demands a certain level of abstraction. Furthermore, I believe that we need medium-specific theories of narrative, theories with a conceptual apparatus sufficiently specialized to define the actual differences between narratives in the various media. In addition to this, we need to be aware of the differences between the two types of narrative theories. The comparative study of narrative in different media, either at the same time or one after the other, is one of the most efficient ways to expose both the common narrative features and the medium-specific aspects of the objects of study. (Hausken 2004: 397)

While I recognize the great value of “medium-specific theories of narrative”, my aim is to form a narrative theory that is “independent of medium” and to systematically chart its components to facilitate detection of also “medium-specific aspects”. I believe that the most efficient way of doing this is to anchor the conceptualization in a semiotic framework. While much classical narratology has been influenced by linguistic semiotics, especially the work by Ferdinand de Saussure, this is a blind alley for transmedial narratology. Language is only one, albeit very important, means of communication, and in my view all attempts to understand communication at large through essentially linguistic theories are doomed to fail. This is partly why semiotics has gained a bad reputation among some scholars. On the other hand, while the semiotics of Charles Sanders Peirce certainly has its pitfalls, it offers a foundational conceptualization that makes possible a truly transmedial understanding of communicative phenomena. Thus, I will use Peircean semiotics combined with some established theories of cognition and psychology. I will also apply some of my own previously developed concepts on communication, intermediality, and semiosis to structure the treatise.

Naturally, I will also profit from achievements in literary theory where narratology has vital roots. Whereas some of its concepts are very useful also for a transmedial approach, many are media-specific rather than transmedial and are therefore not suitable to deal with in a treatise like this. There is also much terminological and conceptual incongruity in the collected body of narratological research, which leads me to avoid some terms that are used also for denoting potentially transmedial concepts, even though they emanate from language studies. For instance, I will not use the central term ‘discourse’, which has been employed in many ways that are rather confusingly interrelated. Instead, the central concepts and distinctions that are associated with the term will be described in other ways.

Thus, I will disregard many ideas and concepts that have been discussed for decades in narratology, sometimes because I find them pointless and often because they are overly media-specific, meaning that they may be highly useful within a more limited frame. It is far beyond the purpose of this study to interrogate these many concepts that I choose to exclude. Instead, I will try to create an account of transmedial narration that is clearly focused on what is most distinctively transmedial, largely but not completely avoiding detours that might make my position within the whole narratological field clearer, but would at the same time distract the attention from my commitments.

I am aware of the risks of such an enterprise. In his insightful discussions of the possibility of forming a truly transmedial narratology, Thon noted that the question remains “if decreasing the granularity of just about any narratological concept until it can somehow be applied to a sufficiently large number of narrative media is, in fact, a good idea” (Thon 2016: 23). Although I will certainly not try to adjust “just about any narratological concept” to fit in the transmedial costume, I expect that some readers of this study will find that my aim of forming a radically transmedial concept of narration is not a very good idea. Of course, I believe that it is; transmedial research sometimes demands not only “a certain level of abstraction” (Hausken 2004: 397) but a very high level of abstraction indeed. Although much traditional flesh will have to be carved off the bones (not necessarily to be discarded, but rather to be remolded), I believe that it is only by starting with the naked transmedial skeleton of narration that one can really detect the similarities and differences among more media-specific narratologies.

Therefore, my ambition is to make this treatise more broadly transmedial than the already existing works on transmedial narration, such as Thon’s (2016) in many ways exemplary and thorough investigation of some central narratological concepts applied to feature films, graphic novels, and video games. Although Thon’s approach is certainly broadly transmedial, it would be quite difficult to include, say, narration in instrumental music or mathematical equations in his framework. A concept such as storyworld, which is central for Thon, presupposes that narratives are understood as representations of concrete, anthropomorphic characters moving around and acting in three-dimensional surroundings. This is not at all a suitable way of conceptualizing narration in media types such as instrumental music—and this is not a far-fetched example, since there is quite extensive research on musical narration from which concepts such as storyworld are largely absent. Regardless of how useful concepts such as storyworld may be for understanding narration in several media types, they are not transmedial enough to find their way into this study.

However, my intention is not only to put aside concepts that are too media-specific but also to embed the investigation of transmedial narration in some central issues of communication at large. Therefore, I will include elaborations on concepts of communication that actually go beyond the defining traits of narration but are essential for making sense of them. Hence, another purpose of this work is to put the issue of transmedial narration in a wide-ranging setting of transmedial communication.

Although I will strive for logical coherence and systematic rigor, and certainly apply many distinctions, I will also resist the tendency to form too rigid conceptual structures. Barbara Herrnstein Smith (1981) criticized the tendency in early narratology to form dualistic concepts (such as story vs. discourse). My own conceptual distinctions should be understood as ways of modeling the complexities of transmedial narration and developing suggestions for methodical thinking about media interrelations. Thus, I wish to avoid trying to define or postulate criteria for what narratives are as such or how narratives in different media types are actually, by themselves, related to each other. If pushed too far, these questions become rather pointless. Narration is a perceived quality that is always somehow grounded in certain media products, and sometimes very strongly so, but is ultimately evoked by the perceivers. Consequently, narrative transmedial interrelations are also, to a large extent, phenomena that emerge in the mind of the perceivers. Therefore, I will interrogate how narratives may be construed by perceivers on the basis of various sorts of media products. A crucial goal is to improve our understanding of how such perceived narrative qualities depend on both basic, material media traits and mental operations that may be either very subjective or strongly intersubjective.

But why bother about transmedial narration at all? On the most general level, it is important to be able to understand and analyze ubiquitous communicative phenomena such as narration simply because communication is so vital for the existence of human beings—and a proper understanding of the place of narration in communication at large requires a transmedial approach. More specifically, a transmedial perspective on communication and narration is necessary for creating links among more-or-less isolated research areas that would profit from cross-fertilization. A transmedial methodology makes it possible to compare, in some detail, how narration works in areas of communication that are, by routine, considered to be unrelated.

Furthermore, developed transmedial concepts enable careful investigations of how narratives are transmediated in all forms of communication in the whole society—from casual everyday communication to advanced political, artistic, or scientific communication—and what the consequences of such transmediations might be in terms of both added and corrupted significance. One vital media characteristic that may be distorted by transmediation, sometimes with immense implications, is truthfulness; coming closer to an understanding of these processes appears to be urgent.

Although it is certainly meaningful and necessary to also investigate media-specific narration, such endeavors will remain incapable of contributing to a broader understanding of narration and human communication at large as long as transmedial narration is not a point of reference.

Disposition

In the remainder of Part I “Drawing the Frame”, narration is first put into the area of communication at large. In Chap. 2, I propose several general concepts for modeling communication and relate these to some influential psychological and cognitive concepts. After that, the stage is set for a transmedial definition of narration followed by semiotic and cognitive elaborations in Chap. 3. Chapter 4 explores the fundamental similarities and differences among media types to explain why different media types may narrate to different degrees.

Part II, “Scrutinizing the Essentials”, systematically investigates the core characteristics of narration and some general forms of transmedial media characteristics that I find essential to framing narration. Chapter 5 suggests a methodical and profoundly transmedial way of analyzing narrators that are external and internal to narratives. Chapter 6 circumscribes the concept of event, central to narration, and Chap. 7 scrutinizes the equally fundamental concept of temporal relationships among events, proposing some distinctions that are vital for grasping media differences. In Chap. 8, the formation of internal coherence in narratives is illuminated from several theoretical perspectives. Chapter 9 proposes some analytical tools for understanding how communication in general and narration in particular can be truthful to what we perceive to be the actual world.

Finally, the brief Part III of the treatise, “Demonstrating the Principles”, illuminates and roughly summarizes some vital concepts and ideas. Its single chapter (Chap. 10) includes four sections investigating narration in dissimilar media types: painting, instrumental music, mathematical equations, and guided tours. These studies elucidate the usefulness of the theoretical framework developed in the treatise and highlight the media similarities and differences that make narration a profoundly transmedial but nevertheless media-dependent phenomenon.