When organizing our understanding of the world around us, we use semiotic resources (e.g. Kress 2010). Semiotic resources are resources that we use to organize our understanding of the world and to make meaning in communication with others, or to make meaning for ourselves.Footnote 1 When these semiotic resources are used in systematic ways they form semiotic modes. Examples of semiotic modes are color (when color is used in more or less conventionalized ways), spoken or written verbal language, images, or gestures. A semiotic mode in itself can actually be viewed as multimodal, for example an image where color is used as one meaning-making resource. Semiotic modes can also be combined differently in different media—such as paper-based or digital media. In all social communication we combine semiotic modes. Thus, social communication is always multimodal (e.g. Jewitt et al. 2016).

1 Knowledge Representations

One might assume that we can choose any available semiotic mode for making meaning, and that it is possible to express “the same” content through any semiotic mode. From such a perspective, the choice of semiotic mode would rather be based on aesthetical or practical choice. A more philosophical question is whether it is actually possible to separate content from form. If we imply that all knowledge must formulated somehow, and that all knowledge is represented through some kind of system for meaning-making, form is crucial for deciding what aspects of the content are given precedence. For an expert of a field, the ways in which content is presented are of minor importance. For the novice entering new knowledge areas, however, the ways in which content is presented can be crucial for the learning process (also see Jewitt 2008; Selander and Kress 2010; Danielsson 2016).

2 What Semiotic Modes “Count” in Schools?

The fact that we can use many ways of communicating content, and that all written texts to some extent are multimodal, can be contrasted to the text traditions in schools, where the written word is given most prominence, as the resource that is still valued and “counts”.

By focusing only on the words in student texts, there is a risk of missing important aspects of what is actually said. In a project on the text worlds of pupils in primary school, Charlotte Engblom (2010) made an interesting point regarding what counts for teachers and pupils, and that their interests can sometimes be contradictory. In one classroom activity, when students were asked to produce a text on the computer, one pupil made a conscious spelling “mistake” on the word mum, to allow the words in the sentence to be placed in a way that made the image appear at the right place on the page. In this case, the visual appearance of the different text resources (image and words) was considered more important than correct spelling. The problem for this student was not spelling, but instead how to handle the digital tool, in this case to make the words appear at the right place without manipulating the word length.

With regard to different views on texts and what counts, traditions vary in different school subjects. In more practically oriented areas, such as crafts or arts subjects, the tradition of viewing the written word as the most important resource is less prominent. At the same time, in such areas there is a risk that the plethora of multimodal texts that are actually utilized might be overlooked, for instance sewing instructions or drawn plans. However, even subjects that we regard as highly theoretical, such as the natural sciences, have long traditions of using an abundance of visual resources and representations, such as images, diagrams, or three-dimensional models.

3 Semiotic Mode and Meaning Potential

One aspect related to semiotic mode is their aptness in relation to different types of content. Spoken or written words are specifically suitable for expressing temporal aspects (first this happened, and then this) or issues of cause and effect (x happened because of y), while images are better apt for spatial relations (e.g. Kress 2003). This does not mean that we cannot use images to express temporal relations; if we want to do that we can use more or less culturally established resources such as arrows, reading order between a series of images, etc.

The choice of resources for meaning-making is linked to the notion of modal affordance (e.g. Gibson 1977; Kress 2003), in other words the “meaning-making potential” or the “potentials and limitations” of certain semiotic resources or modes in a specific situation.

The term affordance can be illustrated by the variety of semiotic resources that students come across when a natural phenomenon like the atom is explained in chemistry classrooms (Danielsson 2016). For example, in textbooks, combinations of writing, including chemical or physical symbols, and images are used, and in their expositions, teachers use words (spoken or written), written symbols, images (drawn on the board or projected on a screen), and gestures. In words, metaphors like “electronic cloud” or “electron shells” are used, or the atom is described through analogies, for instance by comparing electronic orbits with the planets’ (electrons) movements around the sun (atomic nucleus).

The choice of semiotic resource for meaning-making is never made at random. Instead, more or less conscious choices are made, based on available modes or specific resources in the meaning-making situation. The choice is also dependent on what we want to say, and to whom, depending on the affordance of different resources. Also, the choice is reflected by what aspects of a phenomenon a certain representation can be used for (e.g. Danielsson 2016, Tytler et al. 2013). For the novice, the intimate connection between content and resources chosen for representation has consequences for learning: to learn what signifies an atom cannot be separated from learning how the atom can be represented. This connection is commented on by Diana Laurillard thus: “Knowledge technologies shape what is learned by changing how it is learned” (Laurillard 2012, p. 3). She illustrates this statement by an example from a course in Business Studies: when the students are expected to read and write, their analytical competencies are supported, when role play is used, their experiences and their understanding concerning interpersonal relations, and when they are using Excel sheets, experiments and calculations of different economic parameters are supported. The given—and conventional—representations have consequences for what is recognized and assessed as relevant knowledge in the subject.Footnote 2

Thus, different resources have different potentials for meaning-making. The different particles of an atom can be illustrated through a drawn image, and at the same time you get an approximate idea of how the different particles are related: that protons and neutrons form the nucleus, and that electrons revolve around the nucleus. Yet, when we draw an image, we have to decide where to place the different parts in relation to each other, and we have to decide their relative sizes.

A drawing of the atom will always be a simplification, and in fact a falsification, for example since the electrons are so extremely small in relation to the protons and neutrons in the nucleus, something that cannot be illustrated on a limited surface such as a textbook page or a board in the classroom. Furthermore, the electrons have to be placed somewhere. They are often drawn in distinctly delimited orbits, although there are in fact no such distinct orbits. Instead, the electronic shells one talks about in science are more of an illustration used to indicate the plausibility of the position of an electron at a given moment.

You need only a quick look at the illustration in Fig. 3.1 to get a general image of the different particles of the atom, and their approximate relations to each other. On the other hand, through words, it is possible to clarify that the electrons are not placed in clearly defined orbits, and that the exact placement is always arbitrary. Through words and mathematical symbols you could also describe, for example, the relative sizes of protons and electrons. With multimodal ensembles, such as combinations of words, images, and other resources, a relevant (though still simplified) image of the atom as a natural phenomenon can be given.

Fig. 3.1
figure 1

Teacher’s illustration of the atomic model in a secondary school chemistry classroom (from Danielsson 2016, with permission from the author)

Full size image

Already when students start school, they have some level of knowledge—most probably implicit—of what is best said through what semiotic mode. Figure 3.2 is an example of how a Norwegian pupil in first grade uses a combination of writing and image when creating a text after an excursion to the sea (Sjøhelle 2013). In that text, the student has drawn a crab that he found during the excursion. For this student it is obviously an important piece of information that crabs move sideways. However, to depict such a movement in an image is difficult, and he comments on that in writing: “crabs move sideways” (No. kraber går sidlengs). School can benefit from the fact that even young students use different modes in accordance with their affordance when creating their own texts and when working with pedagogic texts in the classroom teachers can connect to this implicit knowledge.

Fig. 3.2
figure 2

Student text (year 1) after an excursion to the sea (Sjøhelle 2013, p. 114, re-printed with permission from the author)

Full size image

Resources used in meaning-making have a culturally established meaning potential which embraces both the offered meaning (for example the intention of the illustrator or the author) and the perceived meaning (what the reader of the text pays attention to). Not the least from an educational perspective, it is important to note that in the actual meaning-making situation it is not always the case that the intended meaning coincides with the perceived meaning. Instead, the reader’s interest, as well as the reader’s previous knowledge in the area, can direct what meaning is actually made from the text.

4 Implications for Education

From an educational perspective, it is a great asset to have an abundance of semiotic resources to choose from when engaging in various aspects of content. Yet we cannot take it for granted that the students will be able to handle the different resources without guidance, or to be able to see their possibilities and limitations. If we stick to the example of the atomic model, we can imagine that for anyone who already knows the structure of the atom, and the characteristics of this natural phenomenon, it can be functional to compare the atom with an apple (an analogy sometimes used in Swedish speaking classroomsFootnote 3) without pondering about what the fruit pulp could be analogous to. Furthermore, the expert knows that there are no prefabricated “shells” for electrons to place themselves upon. However, such analogies can give false ideas to anyone who is not familiar with the content, such as a student who in an interview expressed the idea that “the shell becomes empty” when ions are formed, with valence electrons “leaving” an outer electron shell.

5 Summary

In this chapter, we have discussed semiotic resources in terms of the resources we use to communicate and to organize our understanding of the world around us. When semiotic resources are used in systematic ways, these can be seen as semiotic modes. Examples of semiotic modes are verbal language (written or spoken), gestures, and color (when the color in itself carries meaning, such as red for heat or to depict “stop!”).

What semiotic resource to use—and in which semiotic mode—is connected to the content that is communicated: you cannot say exactly the same thing through any semiotic resource. Semiotic resources or modes are said to have different affordances, or potentials for meaning-making.

The ways in which a text is designed through different semiotic resources are particularly important for the “novice” who is about to enter a new domain of knowledge. From an educational perspective, we cannot take it for granted that our students are able to handle the different multimodal resources without guidance, or to fully understand their meaning-making potential (or affordances and limitations). By working actively with the students and making visible to them how texts are structured, we can support them in reading the texts as genres of texts in specific content areas, and consequently make it possible for them to deepen their knowledge in the particular subject content.

The extended text concept as well as the understanding of multimodal texts challenge the dominant school tradition, where writing has been seen as the main resource and the resource that is valued and “counts” as representation of knowledge. It also challenges our more traditional ways of assessing knowledge primarily by the use of “written, verbal language”.