Keywords

Introduction

It is well-investigated, that learners use different modes in mathematical interactions which are understood as a central space of mathematical learning (Huth, 2022; Krummheuer, 1992). The use of different modes is often framed as multimodality in mathematics education (Arzarello, 2006; Radford, 2009). It refers to an integrated use of modes to construct the interactionally negotiated mathematical content. Radford (2009) describes “[…] that mathematical cognition is not only mediated by written symbols, but that it is also mediated, in a genuine sense, by actions, gestures, and other types of signs.” (p. 112). Additionally, in line with C. S. Peirce’ concept of signs and diagrams mathematical learning is described as a socially grounded and visible activity on and with signs and diagrams (Dörfler, 2006). This assumption picks up Peirce’ description of diagrammatic reasoning as the core of doing math with others, even in a very early stage of mathematical development. In this paper, especially actions and gestures of a kindergartner are considered in relation to the simultaneous speech while participating in an open-designed mathematical interaction. In the course, the focused child Rigon develops different mathematical diagrams: First, he places blue and green wooden dogs in a disorderly crowd in front of him, then he creates a patterned row out of these dogs. His gestures, actions, and speech address different mathematical ideas sometimes in an interwoven multimodal way, but sometimes rather in a parallel process. In the following, mathematical learning is considered diagrammatically, the research focus and the used method are described, and different sequences of Rigon’s diagrammatic work are analytically considered and summarized in the conclusions.

Theoretical Background – Mathematical Learning as Diagrammatic Reasoning

In a semiotic view, mathematical reasoning can be characterized as the use of signs and diagrams (Hoffmann, 2003). Both concepts are understood in the sense of Peirce. A sign after Peirce is a triadic relation of a perceived sign (representamen), the referred meant (object), and the effect of the sign in the mind of a sign reader (interpretant) (Peirce, 1932, CP 2.228). Thus, anything can become a sign when it is perceived as such. The representamen can be understood as the perceivable sign, e.g., an action or a gesture, even though Peirce also refers to exclusively mind-based sign processes. The object is describable as what is assumed by the sign reader to be meant by the sign creator (Schreiber, 2013). It is not necessarily a thing or something materialized, but can also be an idea or a statement. The interpretant is a kind of impact on the mind of the sign reader. It is not a person, but an effect of the sign stimulated by perceiving something as a sign. It can be expressed as a new representamen, interpreted again and so on, which refers to the infinity of the sign process. Diagrams, as complex signs, can be seen multifaceted: rule-based written, arranged with material (Peirce, 1933; Billion, 2021; Dörfler, 2006; Schreiber, 2013), or even gestured as a fleeting diagram (Huth, 2022). They can evolve out of rule-related inscriptions, a concept based on Latour and Woolgar (1986). A line, as a first inscription, can be diagrammatically interpreted as a side of a square, a part of a tally sheet of data collection, or as a straight line, etc. Following Peirce, the construction of a diagram, the observation of diagram relations and the manipulation of that diagram to prove if the socially established rules “will hold for all such diagrams” (Peirce, 1931, CP 1.54) are the core of doing mathematics. Diagrammatic reasoning needs to be constantly done with others, especially in early education. Thus, interactionally negotiated and shared meanings and the use of mathematical diagrams are the results of mathematical interactions, offer learning opportunities, and potentially new insights into mathematical relations.

Diagrammatic Reasoning with Gestures and Actions

Previous research on the diagrammatic reasoning of learners shows, that their actions and gestures can be used to reconstruct their diagrammatic interpretations (Billion, 2021; Huth, 2022). With actions, materials can be arranged rule-based and mathematical relations can come to the fore. These are based on that materialized construction and the reading and use by learners as a diagram. Gestures show different functions in mathematical learners’ interactions, e.g., they can link different material parts, show diagram manipulations or even be used as diagrams themselves (Huth, 2022). Vogel and Huth (2020) investigate interfaces of gestures and actions in function, meaning and chronology which refer to the special interconnectedness of gestures and actions in mathematical learning. These results prove that gestures and actions are of great importance in learning mathematics and open up new opportunities to understand the learning process as a multimodal emergence. Gesture researchers agree that the interplay of gestures and actions is complex in daily interaction and that these modes show a special relation to the speech used (e.g. Harrison, 2018). A clear demarcation between action and gesture is difficult and probably rather a matter of theoretical perspective. Some gesture definitions seem to provide a clear distinction at first glance, e.g., that of Goldin-Meadow (2003) who distinguishes a movement to communicate from a functional act on objects. But this turns out not to be tenable, because the material can be integrated into gestures and actions can show communicative intention (Huth, 2022). Andrén (2010) describes two perspectives to be found in the literature, where the first ascribes nearly every body movement to gestures, like hands, arms, and head movements, but also actions, gaze or mimic expressions. The second perspective is more narrowed and in line with Kendon’s (2004) definition (Andrén, 2010, p. 11): Kendon (2004) defines gestures to be interpreted from a counterpart as movements with “features of manifest deliberate expressiveness” (p. 15). He describes gestures as “visible actions” (p. 7). Following Kendon (2004), actions and gestures can rather be distinguished by their function and usage due to the mathematical context and sign interpretation. And sometimes it will still remain uncertain. Harrison (2018) focuses on the commonalities of actions and gestures more than distinguishing features, so he clarifies that across an interaction, actions can become gestures, which however always refer to the action made at the outset. Gestures are performed according to the material world, and “may require elements in the physical surround as an integral part of their semiotic structure […]” (Harrison, 2018, p. 161). E.g., a pointing can be made at a material arrangement and by using an object like a pencil to broaden the pointing itself. As discussed in Huth (2022), the differentiation of gestures and actions cannot be purely made by the question of material use or not. It is more to be seen on a kind of continuum where action and gestures meet at the pole of material use to change an arrangement (action), or material is used while performing a gestural utterance. Actions cannot be ascribed purely functional features but also a potentially communicative intention when made in interaction. It always depends on the interactional interpretation and use of these expressions.

For the present paper and in line with the above discussed perspectives, it can be assumed, that actions and gestures show comparable structures, features and forms. Furthermore, they show comparable relations to the simultaneously uttered speech (Andrén, 2010; Harrison, 2018).The definition as and the meaning of a gesture or an action depends on the overall activity of which they are a part and thus also focuses on usage in attempting to reconstruct the meaning and function of an action or gesture in interaction (Andrén, 2010). Describing gestures and actions in that way fits in line with the thoughts of Wittgenstein (1984), who states that words only gain their meaning in usage. Transferred to actions and gestures, their meaning is reconstructed in their use in the mathematical occupation.

For the analysis in this paper, a distinction between actions and gestures of the learner is operationalized as following: Actions on material arrangements generate new material arrangements which then exist independently of these generating actions (Billion & Huth, 2023). These new material arrangements can potentially be interpreted as a diagram in a semiotic sense. Gestures can be generated on these material arrangements and can indicate manipulations of the material (Huth, 2022). However, the display of the manipulation is not fixed in a new material arrangement and is thus ephemeral (Billion & Huth, 2023). Therefore, actions and gestures have a certain proximity, because a gesture can indicate manipulations, whereas an action can perform the manipulations and fix them in a new material arrangement (Billion & Huth, 2023).The multimodality in mathematics learning is seen as an integrated view of gestures, actions, and speech (and potentially other expressive modes) which show a specific relation to each other and which are interpreted as potentially meaningful for the mathematical occupation based on their interactional usage by the participants.

Research Focus

In relation to the discussed view on gestures and actions and their interplay in mathematical learning understood as diagrammatic reasoning of learners, this paper aims to identify how a young learner (Rigon) uses these expressive modes in relation to his speech while doing mathematics. As a first analytical approach, the interaction in the group is considered, building on this, Rigon’s gestures and actions are focused in relation to his speech based on a semiotic perspective on mathematics learning. Rigon’s usage of actions, gestures, and speech, while he is obviously following his own mathematical interpretations in a complex networking with the surrounding situation and his counterparts in interaction, is of main interest to clarify his multimodal diagrammatic reasoning in a very early stage of learning. From the theoretical considerations, the following research question arises: How are especially actions and gestures in their interplay concerning speech used by a young learner Rigon to create a mathematical idea of determining the number of countable things in the interaction with others?

Methodology

The considered example is taken from the longitudinal study erStMaL (early Steps in Mathematical Learning), which investigated the mathematical development of learners from kindergarten to the second year of primary school in different situations with the potential for mathematical discoveries (Brandt & Vogel, 2017). Another part of the chosen example is analyzed in Billion et al. (2020) and in Brandt and Krummheuer (2013).

Data Generation and Method of Data Analysis

The situation analyzed in parts can be assigned to numbers and operations. For the implementation of the situation with four kids in a German kindergarten, the accompanying person was trained with the help of “mathematical situation patterns” (Vogel, 2014, p. 232) to support uniformity in implementation. The considered situation was videotaped and transcribed using a special notation of all gestures and actions uttered simultaneously with speech (Huth & Schreiber, 2017). The method of analysis combines the interactionist and the semiotic perspective on mathematical learning. Based on the transcript, an interaction analysis in the sense of the reconstructive social research approach was conducted in which the focus is on gestures and actions in relation to speech to generate an interpretation of the ongoing interaction that proves to be plausible (Krummheuer, 1992; Huth & Schreiber, 2017). In the second step, semiotic process cards are created based on the results of the interaction analysis (Huth & Schreiber, 2017). In these cards, the above-described triadic sign concept of Peirce is used to reconstruct the related sign process of actions, gestures, and speech. Due to space constraints, only simplified sections of the semiotic process cards are shown in the following in which the analytical results of the interaction analysis are integrated.

Empirical Example – The Mathematical Exploration Situation

At the beginning of the mathematical exploration situation, four young learners are offered a large number of different wooden animals. The learners investigate the question of whether the number of various animal species differs. All participants sort the unordered quantity of wooden animals according to the different types of animals. To determine the number of wooden animals of a species, various mathematical ideas can be identified, which the learners express in a multimodal way. For the analysis, the actions, gestures, and speech of Rigon (4.7 years) are focused to identify which mathematical ideas he expresses about number determination in these modes. Rigon has chosen the wooden dogs, which are in a disorderly crowd in front of him. In total the 19 dogs consist of green and blue dogs of the same size and shape. First, a sequence is considered, where Rigon wants to find out the number of dogs by counting. Second, he places the dogs next to each other in a gapless line so that he may be able to determine the number by the length of the line. While setting up the series, Rigon has another mathematical idea that deals with pattern sequences. He places the green and blue dogs alternately next to each other, creating a continuous pattern of green and blue, possibly to count them with help of the pattern. In the last sequence, Rigon manipulates the pattern. In the following, these ideas of number determination, including mathematical areas of numbers (counting), measurement (comparing length), and patterns and structures (patterns of blue and green dogs), are analyzed with a focus on the interplay of actions, gestures, and speech.

The Idea of Number Determination by Counting

Rigon first expresses verbally the idea of counting the dogs addressed to the accompanying person. She paraphrases Rigon’s statement and asks all learners to count their animals. Following the request, Rigon says “one” aloud and pushes a green dog a few centimeters to the front left (see Fig. 1, line 1).

Fig. 1
An illustration has 4 stages. A downward arrow at the left reads, sequence 1, counting dogs. From top to bottom, the stages are labeled speech + action, action, speech + gesture, and speech + gesture. Each stage has photos attached to it.

Counting process of the first four dogs

Full size image

With this action, he assigns his phonetic number to a green dog and separates it from the other dogs. It can be assumed that he marks this dog as counted and wants to separate it from the rest of the dogs. He then moves another green dog in the same direction (see Fig. 1, line 2). After this action, he makes a pointing gesture to the latest moved dog and utters “two” at the same time (see Fig. 1, line 3). It can be assumed that Rigon wants to separate the second green dog from the rest of the dogs with his action. However, he marks this one as counted only after the separation from the other dogs, using a gesture. The marking and separation of the first dog is expressed beforehand by an action, that of the second dog successively by actions and gestures. The marking of the dog is now done by a gesture, the separation by an action. In the speech, the counting process is expressed. Through analysis, it is likely that the mathematical idea of a one-to-one assignment and separation of the quantity is expressed first exclusively in action and then with action and gestures. Looking at the further counting process, Rigon abbreviates the interaction of the modes and exclusively uses gestures to mark the dogs and to assign each number word to one of them. He utters “three four” and assigns each of the number words to blue dogs with gestures (see Fig. 1, line 4). For the number word five, he does not make this one-to-one assignment of the word to one dog in the quantity. While he is uttering “five”, he is simultaneously marking two green dogs gesturally. Rigon seems to get confused in counting as he makes more gestural marks than phonetically named ones. Maybe this emerges out of the abbreviated interplay of gesture and action. By pushing the dog away and separating it, he might have assigned only one dog to each number word. Rigon makes a similar assignment with the number word seven as he did earlier with five. He gesturally assigns two dogs to the number word, too, maybe because of the given rhythm with two syllables “sie-ben” in German. Otherwise, he gesturally assigns a blue or green dog to each number word up to 12. After the number word 12, he continues counting with the number word 21 and continues to establish a one-to-one assignment between the dogs in front of him and the number words 21, 22, and 23. Subsequently, his speech becomes slurred and he does not make purposeful pointing gestures to individual dogs, but a fluid movement over the crowd of dogs. He then emphasizes in his speech that he has 12 dogs in front of him. In this sequence, actions, gestures, and speech express the same mathematical idea for determining an unknown number of dogs, even not always successfully. The characteristic color does not seem to play a crucial role. In the sequence gestures and actions are in a close interplay. Rigon creates his trial of a one-to-one assignment multimodally. There is a transition from the usage of actions, to actions and gestures, to gestures, whereas the matching speech is always present.

The Idea of Number Determination by Comparing Length and Pattern Structures

In the second sequence, Rigon seems to determine the number of dogs based on the length of a gapless row. He could be inspired by the interactors who have already placed animals next to each other in a gapless species-wise row. To determine the number of dogs, Rigon first places two green dogs next to each other and then alternates their color. The result is a pattern with the basic unit green-blue. Maybe he changes his idea of determining the number of dogs by the length of the row. The pattern should perhaps help him count them. It remains open which mathematical idea he follows. With a closer look at setting up the dogs, it becomes clear how he expresses the idea of the gapless pattern. In his action, Rigon chooses a green dog and places it to the right of a green dog in the row (see Fig. 2, line 1, right). Maybe he thinks this dog is suitable to be the next one. During his action, the accompanying person asks if another wooden animal is a chicken or a rooster (see Fig. 2, line 1, left). Rigon replies “chickens” (see Fig. 2, line, 2 left). During his speech, he selects a green dog, leads it to the row of lined-up dogs, leads the green dog back to the disorderly crowd of dogs, and lets it go (see Fig. 2, line 2, right). Possibly, Rigon feels this dog is not suitable to continue the pattern. Rigon then selects a blue dog and places it to the right of the green dog. Maybe he thinks this dog fits the pattern. He lines it up without any gaps. Across this excerpt, it can be reconstructed that Rigon’s mathematical idea of setting up a row with a certain pattern is expressed in his actions. How he uses this row to determine numbers cannot be figured out. Nevertheless, he participates in the interaction with his speech, which is unrelated to the mathematical idea he pursues in his actions. He uses the modes of speech and action separately to follow both his mathematical diagrammatic idea of a pattern and the interaction theme.

Fig. 2
An illustration has a downward arrow at the left titled sequence 2, setting up the pattern of dogs. From top to bottom, it has 2 stages. 1. accompanying person is that a rooster or a chicken. 2. gives an answer to the question about the sex of animals. Each stage has photo attachments.

Setting up the pattern of dogs

Full size image

A few minutes later, after Rigon has setup the line of dogs he leads his two hands to the ends of his set up row respectively (see Fig. 3, line 1). Maybe Rigon wants to measure the length of the row with his gesture which is followed by an action. He grasps the far-left dog with his left hand and transfers this dog to his right hand. He places it to the right of the blue dog on the far right of the row (see Fig. 3, line 2). The row that started earlier with two green dogs now starts with a green dog followed by a blue dog. It now ends with a green dog after a blue dog. Maybe Rigon’s gesture shows a mathematical idea to measure the length of the row to deduce the number of dogs. This fits the created rows of the other kids. Possibly Rigon marks the row as finished and now measures it as a whole.

Fig. 3
An illustration of 2 stages has a downward arrow at the left titled, sequence 3, changing of the pattern. From top to bottom, the stages are gesture and action. Each stage has photo attachments.

Changing of the pattern

Full size image

While he marks the length of the row with his hands, he recognizes the ends of the row. He notices that the pattern is carried on differently at the beginning of the row, which he then changes with an action. The mathematical idea of creating a pattern with the basic unit green-blue comes to the fore. A gesture changes into an action, and in the transition, Rigon’s mathematical idea also changes.

Conclusion

The paper aimed to investigate how Rigon’s actions and gestures interplay with speech to create the diagrammatic work of a kindergartener while he constructs mathematical ideas for determining the number of an unknown crowd. The results show that Rigon is likely to focus on different mathematical areas in different modes, implements the same mathematical ideas in different modes, or addresses mathematical and non-mathematical content in different modes.

First, the analysis shows that his focus is on numbers and operations with a counting process. He probably interprets mathematical relations in an already constructed flat arranged crowd. In the semiotic sense, he is likely to create with an interplay of gestures, actions and speech a countable diagram out of a disordered crowd. For this diagram of determining the number of the crowd, the color of the wooden dogs is not essential. The analysis shows that he then changes this diagram in his actions. In the new diagram, he probably establishes two mathematical relations in different modes: In his actions he, first, makes a gapless row to map the length of all dogs. Second, he places the dogs next to each other in a pattern where the color alternates. In his gesture he embraces the row of dogs as a whole, marking the length, Rigon becomes aware of an irregularity in the pattern and changes it with an action. The first sequence shows that gestures, actions and speech can express the same mathematical idea (counting a crowd) and that an action can be replaced by a gesture in the counting process, not necessarily working. This result fits the description by Harrison (2018), who uses everyday examples to show that actions can become gestures in interaction. His results can be extended in this paper because in the case study described here it can be shown that gestures can also replace actions in mathematical interactions. In the second sequence, the analysis shows that the modes are not necessarily interwoven but rather proceed in phases of mathematical interaction parallel to each other. Rigon participates in the interaction involving “chickens” in speech and simultaneously sets up a row of dogs with a specific pattern in his action. Regarding Radford’s (2009) quote at the beginning of this paper “[...] that mathematical cognition is not only mediated by written symbols, but that it is also mediated, in a genuine sense, by actions, gestures, and other types of signs.” (p. 112), it becomes clear concerning the second sequence that actions, gestures and speech interplay, but not necessarily in relation to the same (mathematical) topic. Thus, multimodality in mathematical learning is not always to be understood in a ‘genuine sense’. In the third analyzed sequence, Rigon probably expresses different mathematical relationships in different modes. He is likely to focus on the pattern sequence in action and on the row as a whole in gesture (geometric length).

Theoretically, these findings show that multimodality in mathematics learning is not only characterizable by an interwoven construction of mathematical expressions, but used to participate in an ongoing interaction, potentially not always about mathematics, and simultaneously to pursue own mathematical ideas, construct diagrams and focus on different mathematical relations.