Keywords

Texts as a Multiplicity of Signs

Every child lives in a multimodal world. Usually, children discover the power of different modalities in speech and drawing, sculpturing, or constructing designs even before going to school. School curricula, however, generally concentrate on reading and writing. In them, the arts commonly play a secondary role. Multimodal educators consider language to be very important, but not the main or the only way for humans to communicate. Education oriented toward the word, spoken or written, is monomodal. Multimodal education, by contrast, is based on the assumption that the literacies of different modes of communication are equally important in learning. Educators who take this approach ask how the visual arts can serve as a bridge to reading and writing and how music and movement can contribute to our expression of meanings and self. In this view, every text is a multiplicity of signs: As a consequence, writing is both a linguistic sign and a visual one.

Walsh (2009, 126) argues that the technological landscape of the twenty-first century has changed: Written text is no longer the most significant cultural tool deployed to shape our social attitudes and beliefs. Unlike many of their teachers, today even young students may develop literacy competencies in multimodal digital and media environments: This allows them to constantly reconfigure the representational and communicational resources of multiple modes through multimodal design (see Scolari et al. 2018). Yet, classrooms still remain primarily entrenched in print literacy pedagogies. Few spaces exist in schools where multiliteracy curricula are enacted, requiring students to critically read or view and design both print and digital texts, harnessing the multiplicity of semiotic systems. The Cultural Literacy Learning Programme (CLLP) challenges monomodal approaches and enables different trajectories of multimodal learning by using visual narratives, talk, play, performances, video-making, and drawing tasks alongside written texts and writing tasks.

Educators implementing multimodal education generally base their approach on semiotics. Within semiotics, the concept of text can be understood as including different modes of communication besides mere writing (Barthes 1977). As Crafton, Silvers, and Brennan (2009, 33) note:

Semiotic theory expands our understanding of literacy and communication by gently sliding language from its central position to work alongside other semiotic modes, particularly the arts, with greater parity. Semiotics is the study of signs, how acts and objects function as signs in relation to other signs in the production and interpretation of meaning. Working together, multiple sign systems produce “texts” that communicate ideas. Texts can take a number of different forms (written, spoken, painted, performed, etc.) but within each text, it is the complex meaning-relations that exist between one sign and another that breathe life into the communication event.

Various scholars investigate multimodal education. In one of these studies, Maine explores how children construct meanings jointly by interpreting various texts through dialogue. The children in her study discussed films, books, and pictures. Maine (2015, 14) describes reading as “a meaning-making process, a co-constructive comprehension event which necessarily hinges on the interaction between children discussing texts together, and also on the way they interact with the texts themselves.” Similar to the semiotic concept of text, “reading” is understood here as communication in other modes besides the written word. While the technical codes in the different modes of texts are different, many narrative features transcend them, and readers draw on many of the same strategies to make meaning from them (Maine 2016, 3–4):

To comprehend the text more fully, we predict what is going to happen, we ask questions of the texts to explore meanings, we empathize with the characters and imagine ourselves in the story, and we make connections to situations we know, or to other stories that we have encountered. This is the same, whether we are reading a film or reading a book, we just use different “clues” to support our mental image of meaning.

Halliday sees human learning as essentially meaning-making and thus as a semiotic process. For him, “the prototypical form of human semiotics is language. Hence the ontogenesis of language is at the same time the ontogenesis of learning (Halliday 1993, 93).” Language is vital to communicating meanings and cocreating them with others—and thus meaning-making is central to learning. To use language to make meanings in collaboration, people first need to learn the language system and the common rules of communication and dialogue (see Maine 2015, 17).

Semiotic Meaning-making Categories: Field, Tenor, and Mode

Unsworth bases his research into multimodal semiotics in education on multimodal social semiotics, which stems from the interconnectedness of linguistic and social spheres. In this, he builds on Halliday who claims that “the structures of language have evolved (and continue to evolve) as a result of the meaning-making functions they serve within the social system or culture in which they are used” (Unsworth 2020, 6). Halliday emphasizes that language is only one semiotic system among many, including artforms such as painting, sculpture, music, and dance, and other modes of cultural behavior not usually classified as art, such as modes of dress or structures of the family. All of these modes of meaning-making interrelate and their totality might be thought of as a way of defining a culture (Unsworth 2008a, 1). Unsworth (2020, 2008b) suggests that all semiotic systems can be grouped into three main categories, which he calls metafunctions: Representational/ideational, interactive/interpersonal, and compositional/textual. These three categories of meaning-making or metafunctions are related to three situational variables that operate in all communicative contexts: Field, tenor, and mode.

In Unsworth’s account, “field is concerned with the social activity, its content or topic,” “tenor is the nature of the relationships among the people involved in the communication” (Unsworth 2020, 6), and “mode is the medium and channel of communication” that is “concerned with the role of language in the situation – whether spoken or written – accompanying or constitutive of the activity, and the ways in which relative information value is conveyed” (Unsworth 2008b, 379). These three situational variables resonate in Kress and van Leeuwen’s (2001, 2006) social semiotics. According to the “grammar of visual design,” images, like language, always simultaneously represent three realities (Kress and van Leeuwen 2006): The material reality, the interpersonal interaction of social reality (such as the relations between viewers and viewed), and the semiotic reality (in which images cohere into textual compositions in different ways).

The situational variables of field, tenor, and mode all appear within the implementation of the CLLP. Field may be detected in the main topics of each lesson and the activities structured around them. In this social embedding, cultural literacy themes such as living together, social responsibility, and belonging—and subthemes such as celebrating diversity, solidarity, equality, human rights, home, social and civic competence, and sustainable development—are used to ignite discussion and to inspire the creation of visual artifacts. Field thus reveals the representational/ideational structures that verbally and visually construct the nature of the events, objects, and participants involved, and the circumstances in which they occur (Unsworth 2008a, 2–3). In the implementation of the CLLP, field was expressed on different levels of abstraction: Starting from a rather abstract problem with an intense social meaning (e.g., social responsibility), a cultural text (a wordless picture book or a short film, usually a cartoon) was explored, serving as a springboard for class discussion and the creation of an artifact, reflected on verbally in captions.

Tenor may be traced in the choice of expression influenced by the social roles that people take in a communicative situation. In the context of the CLLP, tenor is revealed in the roles that the students adopted as viewers and readers of films and books and as creators of their own artifacts, or “texts” (text here referring to images and combinations of image, text, and sound). The assumed audiences that they addressed were teachers and researchers, but also other students. As tenor is affected by expertise, status, gender, and age, one might expect the students to adopt registers that transmit their roles as learners following school conventions and their cultural adherences in general. In order to understand changes in modality, one thus needs to consider whom the students seek to address.

Mode becomes visible in the choice of medium, or in terms of semiotic reality, the choices of expression on word/image level. In this case, a focus on modes zooms in on the expressive means and the conventions followed to communicate the desired idea or effect. As Kress (2010, 28) notes, “in communication several modes are always used together, in modal ensembles, designed so that each mode has a specific task and function.” Speech may combine with gesture, still/moving image, action, and color in whatever way is considered an apt means of representation.

We refer to multimodality as the intertwined use and transitioning between modalities such as written text, image, audiovisual image, sculpture, theater, etc. Yet changes in modality occur also when a written narrative is turned into a poem, or when a pencil drawing is produced by reinterpreting a digitally produced image. That is, different modalities exist within images alone or texts alone as well. Kress and van Leeuwen (2006, 154–174), for example, discuss modality as a means to evaluate the reliability of messages via their “realness.” In this account, “high” modality refers to a higher amount of detail as well as the use/prevalence of perspective and color (following the assumed “naturalistic” objectivity of the photographic image) while “low” modality is defined via the lack of the above, or, flatness and lack of detail and color. Yet, what is more central to our approach is that multimodal expression may include questions of authenticity and authorship (and hence creativity): Multimodal text composition may resort to practices such as downloading, sampling, cutting and pasting, and recontextualization, and thus it is prone to accusations of plagiarism and “mere copying” (Kress 2010, 24). However, as stated before in chapter two, we view similarities in the students’ artifacts as proof of dialogic engagement with the source text and with the artifacts produced by other students.

Tracing Field, Tenor, and Mode (Material, Social, and Semiotic Realities) in the CLLP

One of the positive challenges included in the project design is related to the multiple structures in the field described above. Since cultural literacy was taught via discussions of multiple abstract themes and various concrete materials it is not easy to discuss the resulting student-made artifacts as one combined multimodal narrative of cultural literacy. The question then becomes: How can the correlation between the themes and the activities be ensured? How can teachers and students maintain focus on a single theme, such as living together, throughout a lesson? To succeed in this, educators had to ask themselves: Does the cultural text that the students are asked to explore respond to their understanding of the abstract theme of the lesson? The wordless picture books and films included the richness of signs enabling various topics of discussion beyond the core theme in each lesson of the CLLP.

Compositional/textual meanings concern the distribution of the information value or relative emphasis among elements of the text (Unsworth 2008a, 2–3). As teaching and learning in the CLLP are based on wordless picture books and short films, the language of these texts is mostly visual. The compositional structure of images in them is expressive. For example, the picture book Naar de Markt (To the Market 2017) by Noëlle Smit suggested for students aged 4–7 to explore celebrating diversity (a subtheme of living together) contains several levels of signs of diversity. The question arises: From whose perspective is the visual story created? Each picture showing what is going on at the market reveals the interests of different groups of people. Stallholders want to sell their products so they are advertising them. Customers want to buy the best food and are watching the sellers and examining the food. Birds want to steal some of the food on display. Yet, the story focuses on a small girl, the only child in most of the pictures. The girl is not interested in the actions of the sellers and buyers since all of her attention is concentrated on the events on the ground: The dogs or cats who are running nearby and the birds that are eating the fish. In the book, the market is full of life, energy, and colors, which all emphasize the cultural diversity of the scene(s). The illustrations depict people with different ethnic backgrounds, skin colors, and styles of dress. This book, and its imagery of peaceful everyday life uniting people with different interests and ethnic backgrounds into a harmonious whole, can be used to discuss celebrating diversity based on equality and human rights.

In the CLLP, the lesson based on To the Market included three optional tasks for the students. In one of these tasks, the teachers and students were asked to consider the sonic aspects of the scenes via questions such as: What sounds do you hear? What do people say? What languages do you hear? What sounds do animals or objects make? To create these soundscapes, students had to change the semiotic mode of the story from visual to auditory. In another task, students were asked to create a visual response to the story by identifying with the stallholders and imagining selling items at the market. In this task, the instructions directed the students to make a drawing responding to the question: What kind of goods do you decide to put on display and sell? This task was thus based on the same semiotic mode as the picture book. Students’ visual response to the book in their artifacts was to present food items familiar to them, including traditional local or national dishes. Some of the teachers changed the semiotic mode from visual to three-dimensional by replacing the drawing task with sculpting (Fig. 3.1).

Fig. 3.1
figure 1

A sculpture of market stalls with local products created by a German student in the youngest age group

Full size image

Pedagogically, To the Market enables the exploration of different perspectives and routines of everyday life. For instance, the story differs when viewed from the perspective of the girl, her mother, a seller, any of the customers in the crowd at the market, a dog, or even a bird.

The CLLP reveals itself as a dynamic teaching and learning practice emanating from a variety of semiotic modes. For example, in a task on sustainable development, the students were asked to create their own “want” pile (to list what they wanted) and to turn this pile into a mess monster following the book they had just read, Balbúrdia (Shambles 2015) by Teresa Cortez, which described a similar metamorphosis. The ensuing artifacts were then photographed and explained with a caption. This instruction illustrates the multitasking nature of the CLLP and its semiotic objective to transform modes of communication.

Immersing oneself even more deeply in texts (picture books and films in the CLLP), allows one to concentrate on their multiplicity of signs and reflect upon their meanings. Roche proposes in Developing Children’s Critical Thinking through Picturebooks (2015) that the illustrations in picture books should be studied, reflected on, and discussed very attentively, even meditatively, starting from the cover page as a paratext creating meanings for the whole book. This type of study takes a lot of time. Time is needed to explore and find correlations between visual and verbal signs within a semiotic mode: within this, compositional/textual meanings are important. The CLLP did not aim at rehearsing students’ visual literacy as such. However, drawing on visual literacy enabled the teachers and students to make sense of the meanings in the visual stimulus. In the CLLP, students engaged most fully with the compositional/textual meanings of the cultural texts when they were asked to describe a particular sequence of events from the picture book or film in their cultural artifact. Students did this in a lesson on the theme of living together and its subtheme of equality, targeted to the second age group and using the film Isän poika (Papa’s Boy 2010) by Leevi Lemmetty as a stimulus. Students were asked to produce a comic strip showing the father’s and son’s emotions at the beginning, middle, and end of the story and to write a short paragraph explaining their comic strip. Compositional/textual meaning in this case played the main role in recognizing the protagonists’ emotions.

The variety of the tasks in the CLLP changed the structure of tenor. As tenor is the nature of the relationships among the people involved in communication, at the very beginning of the CLLP lessons the students can be considered as viewers establishing their relation to what is viewed. Receiving the task to create the artifact themselves changed the interactive relationship to the readers, writers, and visualizers, and thus as interpreters and meaning creators. This reflects Kress and van Leeuwen’s (2006) idea of relating what they call the “image act” to the system of speech act and person in language. The system of person can describe the tenor as the nature of interpersonal communication. There are three basic options: First person (I or we), second person (you), and third person (he, she, they).

In our data, the students often used the “I” perspective to explore the themes and subthemes. One illustration of this is the artifact with a very short caption in Fig. 3.2, which was created by a student in the oldest age group. On the left side of the picture, one can see the word “Mum” and the name of a city. On the right side of the picture, one can see the word “Dad” and the name of a village. The signs of the picture tell the story of a life split between different spatial locations. The child in the picture is standing alone between the different spaces and their social spheres. The artifact was a response to the task in which the students were asked to create a leporello (a concertina-folded leaflet), with a sequence of sketches representing their own everyday culture in a lesson on living together and the subtheme of celebrating diversity, using the book Excetric City (2014) by Béatrice Coron as a stimulus. The student preferred to work on this topic alone, focusing on her current life situation.

Fig. 3.2
figure 2

Artifacts by Israeli (above) and Lithuanian (below) students in the oldest age group exploring the topic of living together

Full size image

As most of the tasks in the CLLP were designed for small groups or the whole class, most captions were written in the first-person plural; the agent was “we” or “us.” However, students could respond to the same task by expressing different perspectives. Many groups responded (as expected) to the above task by emphasizing the “we” perspective: “We are all different. We painted our celebrations, friends, the gym, school, home – the things which are personally important to us. These drawings indicate our differences,” as one group of students write in their caption. Another group noted: “These drawings indicate our differences because we all think differently, everybody’s attitude to the same aspects is different.” The perspective could also change from “we” to “they,” as a caption by one group of students doing this same task illustrates:

In this book [artifact] we wanted to show the world’s uniqueness and variety. The world on its own isn’t original but people make it authentic by coloring its parts. Each of us colors a little piece and together the world becomes a rainbow full of creativity and rich in its unique beauty. People’s authenticity was shown in the book.

The intertwinement of the “we” and “they” perspectives is also visible in the following caption where a group of students gave their leporello:

We tried to portray that people can help, give to each other when they don’t have something. We all have some emptiness within ourselves and we are different in the way we choose to fill it. This book is trying to express those ways of filling. People who experience the same empathy usually look at it differently. But what matters is what we give, not what we receive.

The caption approaches empathy from a “they” perspective. It was, however, more common in the captions to deal with empathy from the first-person perspective.

The change of perspective and its impact on meanings is illustrated in an artifact by another student responding to the same task of creating a leporello (Fig. 3.2). In it, this student from a little town describes her own daily life. She tells the reader what she likes: Nature and meeting her friends. She ends her caption by changing her perspective from “I” to “we” and challenging the optimistic mood of the previous self-presentation:

In my page, I liked to show that our lives and environment are not always perfect. Some of us enjoy good marks at school, popular friends, but at the same time, we do not always notice that there are a lot of different people, who are not so happy. It’s a pity that we don’t always try to support them, to help them. Even when they experience bullying.

In her expressive picture, one can discern various visual signs of bullying. These signs visualize bullying as a cloud full of mockery and aggressive gestures that one cannot avoid. The cloud spreads over the horizon. The student who experiences bullying is captured in a dark circle. The faces of the other students disappear: They become like stony mannequins, not supporting or helping the classmate in trouble. By visual signs, the creator of the artifact tells the story of bullying and reveals the deep loneliness of an unhappy child. Somehow both the pictures in Fig. 7.2 express more than what can be described in words. Behind the images lurks sadness or even despair, inexpressible in words. When comparing linguistic and nonlinguistic devices, Eco (1976) noticed that both contribute to a subset of contents which are translatable from one device to the other; this conception leaves aside a vast portion of “unspeakable” but not “inexpressible” contents. The “unspeakable” but not “inexpressible” in both alternatives—verbal and nonverbal—always remains (Eco 1976, 173).

To sum up, the learning process in the CLLP is based on multimodal education, in which one mode of communication becomes interpreted and explored through another. For us, multimodality is a “normal state of human communication” (Kress 2010, 1), and every text can be perceived as a multiplicity of signs. We discuss this multiplicity with the semiotic concepts that operate in all communicative contexts: Field, tenor, and mode. Through them, the CLLP can be seen as a space for engaging in social activities; exploring cultural, social, and societal contents and topics; and creating and elaborating social relationships. Various media and communication channels are used to do this, ranging in the CLLP from linguistic to visual and from auditive to performative expression.