Embodied metaphors in tangible interaction design
- First Online:
For centuries, learning and development has been supported by physical activity and manipulating physical objects. With the introduction of embedded technologies, opportunities for employing tangible or embodied interaction for learning and development have emerged. As a result of previous research, we have seen that interaction models based on embodied knowledge (through embodied metaphors) can support children’s learning in abstract domains. Although metaphorical mappings are promoted in tangible and embodied interaction research, little is known about how to identify embodied metaphors, or how to implement them effectively into interaction models. In this paper, we introduce a people-centered, iterative approach to the design of tangible learning systems with embodied metaphor-based mappings. As a design case, we implemented our approach to the design of Moving Sounds (MoSo) Tangibles; a tangible system for learning abstract sound concepts. The system consists of a set of interactive tangibles with which children can manipulate pitch, volume, and tempo of ongoing tones. In a user study with 39 participants, we found that all children were able to reproduce sound samples with MoSo Tangibles.
KeywordsInteraction design and children Tangible user interfaces Embodied interaction Design research People-centered approach Embodied schemata Metaphor Interactive learning systems Musical sound education
In recent years, new interaction styles have emerged, which aim at leveraging human skills in interaction with technology. Many studies (e.g., [30, 32]) in the area of both embodied interaction  and tangible user interfaces  have revealed potential benefits of such interaction styles for learning and development (also see ). O’Malley and Stanton-Fraser  state that tangible systems encourage discovery and participation. Zuckerman et al.  developed multiple tangible learning systems, promoting self-reflection when learning in abstract problem domains. Antle  states that embodied interaction engages children in active learning, which can support cognitive development.
In a previous study , we have explored a whole-body interaction learning system that implements an interaction model based on embodied metaphors; the mapping between action and output relied on embodied metaphors, metaphorical extensions of embodied schemata, which are cognitive structures that are applied unconsciously in learning. This study indicated that the implementation of such embodied metaphors may enable children to reason about abstract concepts in an interactive environment by leveraging or applying embodied knowledge which is formed through early experiences in the physical world.
Incorporating embodied metaphors in learning systems therefore seems promising. However, to effectively support learning through embodied metaphors, successful interaction design is crucial. Not only must one identify the embodied metaphors children use in their understanding of the targeted abstract concepts, these metaphors must also be effectively translated into interaction models and incorporated in interactive systems.
In this paper, we introduce a people-centered, iterative approach to the design of interactive learning systems with embodied metaphor-based mappings. This approach consists of five phases and mainly relies on user involvement during the design research process. As a design case, we implement this approach in the design of Moving Sounds (MoSo) Tangibles, a tangible system for learning abstract sound concepts, and report on the different phases of the design process. Although the design process and the results of this design case are closely connected, the main focus of this paper is describing the approach rather than the specific results. The design case we present builds on and extends previous work  by focusing on tangible interaction rather than whole-body movement. This also enables us to discuss design considerations for both interaction styles regarding learning systems. First, we will look into the theoretical background of embodied metaphors and learning theories as well as related systems in the area of tangible and embodied interaction.
2 Theoretical background
2.1 Embodied metaphors
Such embodied schemata are used to reason about abstract domains. For example, when we say “I am in love,” we (unconsciously) apply the embodied schema in–out to structure our understanding of the abstract concept love, viewing love as a container and ourselves as an entity being in or out of this container. This human ability to project the structure of bodily originating schemata onto a conceptual domain is what is meant by metaphor . A metaphor allows us to understand or experience one concept (target domain) in terms of another (source domain). When the source domain involves schemata that have arisen from bodily experiences, we call them embodied schemata and the metaphors, embodied metaphors.
2.2 Learning theories
Since the beginning of the twentieth century, several psychologists have studied the use of physical objects for learning and development. Vygotsky and Galperin, for example, state that “mental acts originate in material acts” [33, pp 21] (quote translated from Dutch). Bruner  has shown that physical objects play a major role in bridging the abstract and the concrete. Both theories [11, 33] underline the importance of combining experience and reflection. This happens for example when learning about addition with an abacus. In such a case, a child will first start sliding the beads (experience), after which he or she will look at the results and notice the beads being regrouped (reflection). Learning and knowledge acquisition (e.g., gaining a symbolic understanding of the concept addition) takes place when frequently shifting between experience and reflection .
Similar to the role of physical objects in mathematics education is the role of body movement in the process of learning and understanding abstract concepts related to musical sound [24, 27], which are the focus of the interactive learning system developed in our design case. Wessel  for example emphasizes that rich sensory-motor engagement enhances the experience of music. Based on children’s early experiences combining movement and sound perception, Juntunen and Hyvönen  suggest a metaphorical link between body movement and abstract sound concepts such as pitch or volume. This link relies on embodied metaphors, which enable children to understand abstract (sound) concepts in terms of concrete (embodied) concepts. For example, children can understand the concept volume (soft versus loud) in terms of concrete, movement-related concepts (for example, slow versus fast or up versus down). Various movement-related metaphors are used in music education [24, 27], bridging the physical to the abstract. This also enables shifting between experience and reflection, which forms the basis for knowledge acquisition.
3 Related work
3.1 Interactive learning systems
As the field matures, a growing number of studies are exploring the design and evaluation of tangible and embodied interaction to facilitate learning and development (e.g., [29, 32]). An early example of a tangible interface developed for learning was introduced by Resnick et al. . They presented “digital manipulatives,” computationally enhanced toys that enable children to explore scientific concepts in a playful manner. Several other examples of tangible or embodied interaction for learning focus on learning of abstract (mainly mathematical) principles. Zuckerman et al.  describe “Montessori-inspired Manipulatives” (MiMs); technology enhanced building blocks that enable children to physically explore abstract concepts. An example of a MiM is “SystemBlocks”; building blocks that simulate system dynamics. Hashagen et al.  present “Der Schwarm,” a full-body interaction environment enabling children to learn about swarm or flock behavior. Horn and Jacob  present “tern,” a tangible system consisting of jigsaw puzzle like artifacts used to create simple computer programs. Girouard et al.  describe SmartBlocks, a tangible interface designed for exploring the volume and surface area of 3D objects. As underlined in several studies (e.g., [4, 29]), tangible interfaces seem particularly valuable for learning in abstract problem domains by relating abstract concepts to physical experiences or concrete examples.
Tangible and embodied interaction is also a frequently explored interaction style for manipulating sound and music (e.g., [26, 36]). Some of these systems target children, such as “Marble Track Audio Manipulator” , a tangible system for creating musical compositions, and “Pendaphonics” , a large-scale tangible interface usable as a musical instrument and performance tool. Body Beats  uses whole-body interaction to help children recognizing and creating sound patterns. Birchfield et al.  presented SMALLab, a whole-body interactive environment that can be used for several educational purposes, including movement and sound teaching. The previously mentioned Sound Maker system  was designed to study the benefits of embodied metaphor-based mappings in interactive environments for children. The present paper extends this work by introducing a design process for the development of embodied metaphor-based (learning) systems, as well as by presenting a design case focusing on tangible interaction rather than whole-body movement.
3.2 Approaches to designing embodied metaphor-based interactions
The aim of the study described in this paper is to explore an iterative design approach to the design of learning systems with embodied metaphor-based interaction models. An interaction model specifies the mappings between input action and output response. Although metaphorical mappings are promoted by several others (e.g., [22, 23, 37]), little is known about how to identify embodied metaphors, or how to implement them effectively into interaction models for new systems.
Fels et al.  use metaphors in their interface design for musical expression. However, the motivation for choosing particular metaphors is not mentioned. In their design of intuitive interactions, Hurtienne et al.  suggest relying on metaphors that are already documented, or using a systematic user-centered design process, in which metaphors are identified through contextual interviews. This latter method seems sufficient when existing interactions are redesigned. However, the design of new (metaphorical) interactions is required in many cases, such as when aiming at activities for which currently no interactive systems exist. In such cases, an analysis of current interactions is not possible. For interactive learning systems, it is furthermore crucial to identify the embodied metaphors that underlie how we structure and reason about the targeted abstract concepts (i.e., identify metaphors that are used to “make sense”). Relying on documented metaphors can help ensuring a bodily basis for the chosen mapping; however, choosing the most suitable metaphor for a learning system seems difficult. Though some design knowledge is derived from example metaphor-based systems , no literature on specific approaches to the design of interaction models based on these metaphors is known to the authors.
In the Sound Maker , embodied metaphors were identified in consultation with choreography experts and elicited through workshop style pilot studies before they were implemented in a whole-body interaction model. One of the findings of the studies with this system was that the “discoverability” of the mappings, which is the likelihood of participants discovering a mapping by chance, turned out to play a major role in learning to use the system . Clearly, both selecting the right metaphors and implementing them effectively are key to the successful design of metaphor-based learning systems.
4 Design approach
Enactment studies to identify applicable embodied metaphors
Creating low-fidelity prototypes based on embodied metaphors, to explore the input design space
Evaluating low-fidelity prototypes to validate the input design space in terms of affordances which support embodied schematic movements
Creating high-fidelity interactive prototypes with suitable affordances, to explore the mapping between the input design space and metaphorically linked output responses
Evaluating high-fidelity interactive prototypes, to validate the input design space, embodied interactional mappings and output responses
In the coming sections, we will discuss these five phases in detail. In order to illustrate our approach, we will elaborate on a design case, in which we implement this people-centered, iterative approach in the design and evaluation of the interactive learning system Moving Sounds (MoSo) Tangibles.
5 Design case
Extending our previous work , our goal for the design case we present here was to design an interactive system for learning about abstract sound concepts. We developed this system, called Moving Sounds (MoSo) Tangibles, in the context of a research study on how to design metaphorical interaction models as well as on how such systems can support learning. This latter research aim is beyond the scope of this paper. Regarding this agenda, MoSo was designed to enable research, rather than to be directly applicable in a classroom context. However, the approach we propose is applicable to both design and research-through-design processes.
The interaction models incorporated in Moving Sounds (MoSo) Tangibles were based on embodied metaphors. In previous work , we found evidence that in some cases, more than one embodied metaphor was suitable to reason about a particular abstract sound concept. For example, changes in pitch can be understood in terms of low–high, but also in terms of slow–fast schemata. If abstract musical concepts can be understood in multiple ways, implementing more than one embodied metaphor-based mapping in an interactive learning system may benefit the learning process of certain abstract concepts. Compared to a system with a single mapping, a system with multiple mappings could make learning easier, as children can be supported in reasoning about the same concept in more than one way. This may result in a more comprehensive understanding of the concept that is potentially more easily transferable to other contexts. This also corresponds to a frequently used approach of using multiple representations when teaching complex scientific concepts [3, 38]. The research goals of the MoSo design case therefore were (1) to explore whether multiple embodied metaphors were applicable to single sound concepts and to identify these specific metaphors, (2) to explore how these metaphors could be implemented in the design of interactive systems, and (3) to explore how children interacted with such systems.
To enable this study, we designed MoSo tangibles; a set of interactive artifacts in which multiple embodied metaphor-based mappings were implemented to support children in learning about a set of single sound concepts. Similar to the previously mentioned Sound Maker prototype , embodied metaphor-based movements were mapped to sound changes, enabling the children to structure their understanding of each sound concept in terms of movement-related concepts. Unlike the Sound Maker system , the MoSo system presented in this design case relied on movement with tangible artifacts rather than whole-body movement. This provided a clear distinction between different mappings, as each different mapping between movement and sound change is integrated in a different tangible artifact. Furthermore, this enabled us to compare the two interaction styles.
In music education, one of the first learning goals is for children to become acquainted with sound concepts. In consultation with music teachers, we found that starting from the age 4 or 5, children learn about concepts such as volume, tempo, pitch, and timbre. To avoid using language, which could make the concepts too abstract, they are generally first explained in terms of movement. For example, as children listen to a melody played slowly and then one played quickly, they may be encouraged to respond to changes in tempo with changes in the speed of their movement. This activity helps children gain a preliminary understanding of the concept tempo in terms of their experiences of movement. Another beneficial activity, which is not often employed in music education, is to have the music react to the children’s movement. As stated by the music teachers we consulted, having children control the music through movement requires that they have mastered a basic understanding of sound concepts (e.g., pitch, volume, tempo). Typically, this occurs in preschool or kindergarten. We therefore targeted our system to children aged 7–9 who have a basic conceptual understanding of the ways sounds can vary or change. The learning goal for these children is then to gain a more comprehensive understanding of these sound concepts, which includes being able to generate as well as reason about changes in sound parameters. This could be seen as a step toward knowledge transfer to other contexts, such as the understanding of how musical notation (i.e. an abstract symbol system) represents sound changes.
6 Enactment study to identifying embodied metaphors (user study 1)
We will now elaborate on each phase of the proposed design approach. Each phase will directly be illustrated by the design case in which we present the MoSo Tangibles interactive learning system.
The aim of our design approach is to effectively design interactive learning systems that implement embodied metaphors in their interaction models, enabling children to leverage embodied knowledge in their understanding of abstract concepts. First, a specific set of abstract concepts that children are to be taught about by means of an interactive learning system should be laid out. Once the targeted abstract concepts have been selected, the next step is to identify the specific embodied metaphors that underlie how we structure and reason about these abstract concepts. To find empirically grounded evidence for relevant embodied metaphors, we propose to conduct an enactment study with children in the target age group. This means asking children to make up movements with which they enact changes in the abstract concepts one is designing for. The goal of this enactment study is to identify metaphorical mappings between actions and changes in these concepts. These metaphors can be used to inform the development of low-fidelity prototypes in the next phase of the design process. Furthermore, this enactment study can be used to validate the extent to which children already have an understanding of the used set of abstract concepts, which can inform the choice of abstract concepts incorporated in the interactive system.
6.1 MoSo Tangibles design case
In the MoSo design case, we conducted the enactment study (user study 1) with 65 children of 7–9 years old (35 girls and 30 boys). These participants were asked to enact changes in sound concepts. This study is extensively reported in  and will be summarized in this section.
As a result of these exercises, it became apparent that the children in the targeted age group (7–9) had a basic understanding of the concepts pitch, volume, and tempo. They were able to recognize the related sound changes, and some could even name them in terms of their parameter values (low–high, soft–loud and slow–fast). The understanding of the other concepts explored in this study was much lower. As advised by music teachers, concepts of which children do not have a basic understanding should first be taught by reacting to music with movement, rather than manipulating it through movement (as was intended with our interactive system). Therefore, we decided to include only the concepts pitch, volume, and tempo in our interactive learning system.
Results from user study 1 for volume, tempo, and pitch: the identified metaphor types, the embodied schemata they are based on and examples of enactments (number of cases between brackets)
The embodied metaphors described in Table 1 extend the embodied schemata small–big (movements that occupy small or large space), slow–fast (slow or fast movements), quiet–wild (movements performed with low or high energy or low or high force), and low–high (low or high location). For tempo, we only found metaphors based on slow–fast, which we subdivided into “succession” (when a movement was repeated slowly or fast) and “speed” (when the actual speed of the movement was linked to the tempo of the music).
When we look at the metaphors used in previous studies , we see that the metaphor we identified for tempo (slow–fast) matched the one used in the Sound Maker prototype. For volume, the Sound Maker  implemented the schema active–inactive in the mapping for volume, which can either correspond to small and big movements or quiet and wild movements. Despite this similarity, in this study, we have decided to distinguish small–big and quiet–wild as the resulting movements were rather different. Furthermore, both metaphors may in different ways support structuring your understanding of the concept volume; either in terms of small and big movements which is often used by the music teachers we interviewed or as low or weak force, which literally results in soft or loud sound (e.g., clapping with low force versus clapping with high force). For pitch, the Sound Maker implemented the embodied schema near–far mapped to pitch (near corresponding to high pitch and far corresponding to low pitch). Interestingly, in this study, this metaphor was not seen in the children’s enactments of pitch.
The results of this user study (see  for the results of all sound concepts) confirm that children enact multiple different embodied metaphors in their understanding of single abstract sound concepts. When comparing the groups that employed whole-body movement in their enactments to the groups that were given artifacts to move with, we saw no major difference in the observed embodied schemata.
7 Designing low-fidelity prototypes
After the embodied metaphors used by the target group to structure their understanding of the abstract concepts have been identified, the next step is to implement them into interaction models. The goal of this phase is to explore the possibilities in the input design space. This requires iteration in the design process. We therefore propose to develop low-fidelity prototypes that can be moved according to the identified metaphors. These prototypes should enable the intended metaphorical movement, but do not need to include technology. Such low-fidelity prototypes may have generic form so that multiple movements are possible, but these prototypes can also be designed to afford specific metaphorical movements. Building several different low-fidelity prototypes enables efficient exploration of different ways of implementing the selected schemata and metaphors into interaction models.
7.1 MoSo Tangibles design case
In the first design iteration of the MoSo design case, we used the embodied schemata identified in user study 1 as a starting point, as well as the near–far schema that was used for pitch in the Sound Maker , to enable comparison. Based on these schemata, 14 low-fidelity prototypes were created that could each be used to link one or more of these schemata to metaphorically related sound changes.
Some of these low-fidelity prototypes were more generic than others in terms of the kinds of actions they afford. For example, a simple stick-shaped artifact can be moved in many different ways, whereas other low-fidelity prototypes afford a single movement (e.g., rotating). Although the eventual goal is to design artifacts that each have one clear interaction possibility, the more generic artifacts were important in the design process because they may inspire the design process and support exploration of different metaphors children use when interacting with such artifacts.
8 Evaluating low-fidelity prototypes: how affordances support schemata (user study 2)
Having developed several low-fidelity prototypes that can map the identified embodied schemata to the targeted abstract concepts, the third phase involves the evaluation of these low-fidelity prototypes with the target group in a second user study. This enables determining whether the metaphors are implemented in the interaction models of the low-fidelity prototypes in a way that affords the intended movement. Results of this second user study will inform the design of the interactive learning environment.
8.1 MoSo Tangibles design case
Results from user study 2 for pitch, volume, and tempo, for each low-fidelity prototype: the most common movement (in the gray rows) and the embodied schemata evident in the children’s movement with the low-fidelity prototypes (in the white rows). The number of children that performed the enactment, in relation to the total number of children that moved the artifact, is shown between brackets. The embodied schema slow–fast has for the concept tempo been subdivided in slow–fast speed and slow–fast succession (shortened to succ.)
In the analysis of user study 2 (see Table 2), we saw consistent patterns of interactions and enactments of metaphors with some artifacts, but less consistency with other artifacts. For example, the two rotating artifacts (bottom and top left in Fig. 4) were rotated by 17 out of 18 participants when enacting changing volume, and the embodied schema slow–fast was enacted by 16 participants. On the contrary, the stick with beads attached to it (top right in Fig. 4), designed based on the embodied schema low–high, was moved in several different ways to enact changing volume, none of which implemented the low–high schema. Interestingly, most of the children working with pitch did move this artifact low and high, showing that the low-fidelity prototype does afford low and high movements. This may indicate that a metaphor extending the low–high schema may be less appropriate for volume when implemented in a tangible artifact, even though it was identified in enactments of changing volume in user study 1.
As mentioned before, the schema near–far was not seen in enactments during user study 1, but was used in the previously performed Sound Maker study . Some low-fidelity prototypes were therefore based on this schema, and many children made near and far movements when enacting pitch with these prototypes (see Table 2). In user study 2, the near–far schema was seen even more often in enactments of changing pitch than the schema small–big. This could be related to the affordances of some of the objects. On the other hand, although near–far is location based and small–big is movement based, the two metaphors are rather similar and could even easily be confused. Holding your hands close to each other and gradually moving them away from each other is clearly based on a near–far schema. However, when one is jumping or moving his arms up and down simultaneous to the near and far movement, possibly as a reaction to the rhythm of the playing music, this same movement could also be interpreted to be based on a small–big schema. This means that the small–big schema that became evident for pitch in user study 1  may in a number of cases actually have been a near–far schema or a combination of both. This may explain why we saw many small–big enactments for pitch in user study 1, but hardly any in user study 2. The fact that we saw quite some near–far enactments in user study 2, but none in user study 1, may also indicate that the interpretation of some of the movements was not consistent between user study 1 and 2. This will be further discussed in the discussion section.
9 Designing high-fidelity prototypes: Moving Sounds Tangibles
Once low-fidelity prototypes have been evaluated, the results can be used to inform the final design of the embodied metaphor-based interactive system. This involves determining which metaphors to implement (based on the results of user study 1) and how to implement them in terms of affordances (based on the results of user study 2).
9.1 MoSo Tangibles design case
The aim of the design case described in this paper is to design a tangible learning system to enable research in the area of embodied metaphor-based learning systems. For the purpose of this research, we decided to select three mappings for each abstract sound concept, which were realized as interactive tangible artifacts forming the learning system “Moving Sounds (MoSo) Tangibles.” The design of MoSo Tangibles will be described in this section.
The tangible artifacts depicted in Figs. 6, 7 and 8 together form the interactive learning environment Moving Sounds Tangibles. As seen in these figures, the artifact “rotator” is used for all three sound concepts. Resulting from user study 2, a rotation movement seems a very clear and sensible way to map the schema slow–fast to sound changes. This metaphor turned out to be applicable to all three concepts. As time constraints did not allow designing three different rotating interactions, we decided to use the same artifact for all three purposes. Given the research aims of this design case however, it would have been ideal to have three different interaction models for rotating movement, as this would enable equal comparison to other artifacts. However, if the aim is to design interactive artifact for classroom use, one may choose to use only one artifact for multiple purposes, if metaphorically feasible, to reduce the number of artifacts in a set.
The Moving Sounds tangible artifacts contain basic sensors that measure the movements that the artifacts were intended to evoke. For example, the rotator contains sensors to measure rotation speed, and the squeezer contains a pressure sensor to measure applied force. The sensor data are wirelessly transmitted to a computer and processed by a specifically designed program (written in Processing ). This program determines the appropriate change in pitch, volume, or tempo and generates sound accordingly. To enable clear perception of the changes in sound parameters, we used basic tones rather than complicated melodies. The technical implementation of MoSo Tangibles is described in detail in .
10 Evaluating high-fidelity prototypes (user study 3)
When a (set of) working prototype(s) with embodied metaphor-based interaction models is available, a third and final user study can be set up in order to evaluate the design. The set up of this experiment may largely depend on the intention of the design. However, we propose assessing how easily users learn how to use the design for the intended purpose. This will reveal how successful the implementation of embodied metaphors in the interaction models was.
10.1 MoSo Tangibles design case
To evaluate how well children were able to interact with high-fidelity MoSo Tangibles, we performed a third user study, for which we recruited 39 participants (age 7–9, 25 girls and 14 boys) from two different elementary schools. The participants were divided over two conditions. Children in the one-artifact-condition played with one MoSo Tangible to learn about each of the concepts pitch (either puller, stick, or rotator), volume (either squeezer, waver, or rotator), and tempo (either accordion, rotator, or shaker). The children in the three-artifact-condition were given all three artifacts to explore each concept. Note that the choice of these conditions is related to the research agenda for which we developed MoSo Tangibles, which also included a comparison of learning effects between the two conditions. However, as this learning-analysis is beyond the scope of this paper, the two conditions do not play a major role in this section. Here, we aim to evaluate the extent to which children were able to effectively interact with MoSo, in order to validate the design of our high-fidelity prototypes. Furthermore, we wanted to compare these results with the results of our previous study in which a whole-body interaction environment was used.
Exploration (3 min): Each child had 3 min to explore one (one-artifact-condition) or three (three-artifact-condition) MoSo Tangibles to manipulate either pitch, volume, or tempo. No explanation was given regarding how to move the artifact(s), or which musical parameters to manipulate. The child was only told that moving the artifact(s) would cause the sound to change.
Three reproduction tasks (3 min): After the exploration, the child performed three tasks in which he or she was asked to reproduce a sound sample with (one of) the explored tangible artifact(s). The children in the one-artifact-condition used the same tangible for each task, while the children in the three-artifact-condition were given a different tangible for each task.
Interview (three-artifact-condition only, 1 min): the children in the three-artifact-condition were asked which of the three tangibles they preferred, as well as which one they thought fit the related sound change best.
These activities were repeated for all three sound concepts.
For each sound concept (pitch, volume, and tempo), three different interactive tangibles are available. To have objective results, these artifacts were equally divided over the one-artifact-condition sessions (e.g., one-third of these children used the waver for volume, one-third used the squeezer, and one-third used the rotator). As this turns the one-artifact-condition into three separate conditions, we assigned 27 children to the one-artifact-condition (9 for each tangible) and 12 children to the three-artifact-condition. The order in which the sound concepts were explored was counterbalanced over the different sessions. The same holds for the order in which the tangible artifacts were handed to the children in the three-artifact-condition.
The user studies were either performed in a separate classroom or in the school’s auditorium. In both cases, no other children but the one participating was present. The user study was captured on video.
From the video taken during the study, we analyzed whether the children succeeded in reproducing the sound samples using the MoSo Tangibles. In this analysis, we found that all children were able to reproduce the sounds within the set timeframe of 1 min, although some additional explanation was needed in approximately 25% of all cases. Such explanation did not involve telling the children what to do but consisted of giving hints such as “what did you hear in the sound sample?” or “how did you move the artifact before?” Therefore, all children based their interactions with MoSo on their own reasoning.
The relative number of times additional explanation was required did not differ greatly between the two conditions; it seemed more related to the individual child (possibly depending among other things on attention span). In other words, even though the children in the three-artifact-condition only had 1 min to explore each object while the children in the one-artifact-condition had 3 min, both groups were equally able to execute reproduction tasks within the set timeframe.
Apart from the video analysis, other qualitative results were gathered from the interviews taken in the three-artifact-condition. In these interviews, the participants were asked to indicate, for each sound concept, which of the three artifacts they thought fit the sound change best (i.e. which mapping made most sense to them). As a result of these interviews, we have seen that the personal preferences were approximately equally divided over the different artifacts (e.g., for volume, five children preferred the rotator, four children choose the squeezer, and three children thought the waver fit the sound change best).
The results from both the reproduction tasks and the interviews revealed that different designs were equally effective. This provides evidence to validate successful implementation of the selected embodied metaphors in tangible artifacts, developed through an iterative and people-centered design process.
We have shown that our people-centered, iterative design and evaluation approach used in the design of an embodied metaphor-based learning system (MoSo Tangibles) was effective in both identifying and implementing embodied metaphors. In this section, we will discuss what we have learned from the implementation of this approach and what we can generalize in order to inform other researchers and designers. Furthermore, we will discuss a comparison with previous work.
11.1 Identifying and selecting embodied metaphors
The design approach described in this paper consists of five phases and relies mainly on user involvement during several iterations. In the enactment phase (user study 1), children enacted sound changes either through whole-body movement or with a generic and non-interactive artifact (a plastic ring). In both the low-fidelity evaluation (user study 2) and high-fidelity evaluation (user study 3) phase, newly designed artifacts were used. When comparing the results of these different studies, we see that design constraints (e.g., generic versus specifically designed artifact) are of great influence on the results of such studies. As mentioned before, the near–far schema was not identified in user study 1, but nevertheless, we implemented it in artifacts evaluated in user study 2 as well as in our final design. The near–far schema was also used in the (whole body) interaction model of the Sound Maker , where children worked in pairs. This indicates that “restricting” the children to use one artifact (the ring) or one body (their own body) can result in different metaphors than the ones found when specifically designed artifacts are used or when multiple artifacts or bodies are involved in the exploration. Enactment studies (user study 1) may benefit from offering participants several different materials to perform the enactments with (e.g., use multiple artifacts, encourage collaboration).
Furthermore, the difference in results between the enactment study (user study 1) and the low-fidelity evaluation phase (user study 2), particularly regarding the near–far schema, may have resulted from a coding mistake in user study 1. The near–far schema has likely been interpreted as a small–big schema in a number of cases. Many children made jumping movements in their enactments as a reaction to the rhythm of the music, which may have caused near and far movements (e.g., with the hands) to appear to be small and big movements, while such movement could indicate a combination of the two schemata. This may explain that near–far was not found in the enactment study, whereas it was found multiple times in the evaluation of low-fidelity prototypes. This coding mistake could have occurred because we used open coding, meaning that we did not set up a coding scheme in advance but clustered the movements during the analysis. Semi-open coding (i.e., pre-defining a number of likely embodied schemata while leaving opportunity for identifying new schemata) could likely have solved this problem. Enactment studies aimed at the elicitation of embodied metaphors could therefore benefit from a list of embodied schemata, such as the one proposed by Hurtienne et al. . This way, one would rely on documented mappings, ensuring a bodily basis for the chosen mapping. Furthermore, direct user involvement as well as an opportunity to identify new metaphors ensures choosing the metaphor that underlies how the target group reasons about the abstract concept, which is particularly challenging when new interactions are designed rather than existing interactions being redesigned.
Although the database proposed in by Hurtienne et al.  does likely not contain all possible schemata and is still under construction, it would be interesting to compare the embodied schemata implemented in MoSo to the documented schemata. When doing this, we see that all our schemata correspond to documented ones (although low–high is documented as up–down), except for quiet–wild. However, the schema strong–weak is documented, which seems rather similar to the way we have implemented quiet–wild, namely as applying weak force versus applying strong force. Furthermore, the small–big schema is included in the database in the “attribute” category, referring to small and big as a property of an object or entity rather than as a quality of movement. Comparable movement-related schemata are not documented in . A possible explanation is that this database  documents image schemata rather than embodied schemata. Although these two types of schemata are comparable and an overlap may exist, the major difference is that image schemata are often identified through linguistic analysis (also see [25, 28]) while embodied schemata arise from bodily experiences. Therefore, movement-related schemata may be documented as embodied schemata (e.g., ) and not as image schemata. This illustrates that leaving an opportunity for identifying new schemata in an enactment study such as user study 1 is essential. However, this exercise has also shown that semi-open coding using a coding scheme based on documented schemata (e.g., ) may help distinguishing schemata that may otherwise be confused. Furthermore, such an approach encourages using commonly known names for the identified schemata, which is useful to the generalization of gained knowledge.
The approach we applied differs from other approaches to the design of metaphor-based interaction models described in literature [6, 14, 22], because we involved users at several stages of the process. In our previously performed Sound Maker study , we mainly relied on experts to select the appropriate metaphors. When comparing the two studies, we see an interesting difference regarding the near–far mapping for pitch. In our presently described study, children mapped near to low pitch and far to high pitch, while it was implemented the other way around in the Sound Maker. Clearly, different approaches may lead to different implementations of embodied schemata. Multiple iterations can help finding the implementation that makes most sense to the target group.
11.2 Using this approach in other contexts
In this paper, we present a people-centered iterative approach to the design of embodied metaphor-based learning systems. In the MoSo Tangibles design case, we involved over 150 children. Though large numbers of participants will increase the reliability of the results, it also complicates and lengthens the design process. Therefore, involving such large numbers of children will not be feasible in all contexts and for all purposes, particularly in design rather than research-through-design contexts. In this subsection, we discuss how this approach can be applicable in a broader context.
In the MoSo design case, a system was developed in which three different musical concepts could be manipulated, each through three different metaphorically related schematic input actions. Given our research-through-design focus, our goal was not to design a system that is directly usable in a class-room context for a particular learning goal. Therefore, the design process started with eight musical concepts. Not all participants in user study 1 were subjected to all concepts. Furthermore, due to the large number of low-fidelity prototypes resulting from our approach to implement multiple metaphorical mappings for each concept, each participant in user study 2 was only subjected to one of the three concepts. Therefore, we had about 13 (user study 2) to 25 (user study 1) participants for each musical concept (pitch, volume, and tempo). In designated design processes, however, it is likely that the number of targeted abstract concepts is clear from the start of the process, as is the number of metaphorical mappings needed. As these numbers are likely to be lower than was the case in the MoSo example, fewer participants would in many other studies be required to come to similar results. Eight to ten participants for each abstract concept or for each metaphor (one participant can work with multiple abstract concepts and metaphors in an evaluation session) are likely to result in sufficient knowledge to inform the next phase in each of the three user studies.
Though fewer participants may be required, setting up and performing three user studies may not be realistic in many design processes. The approach we propose consists of five phases: (1) identifying embodied metaphors, (2) creating low-fidelity prototypes based on these metaphors, (3) evaluating the implementation of metaphors in these prototypes, (4) creating high-fidelity prototypes, and (5) evaluating these prototypes. Though all these phases are important to successful design of metaphorical mappings, we suggest that particularly evaluating which metaphors the target group may use in their understanding of the abstract concept in question is an essential part of the design process. When the approach we propose is to be shortened, one could integrate the identification of embodied metaphors and the evaluation of low-fidelity prototypes into one user study, which could be comparable to the study in the low-fidelity evaluation phase (user study 2). This would require selecting embodied schemata that may likely be metaphorically appropriate from literature (e.g., ). These schemata can inform the development of low-fidelity prototypes, which are to be evaluated through a user study. This user study should then primarily aim at validating the selected schemata and secondarily on the evaluation of the designs.
The approach we propose in this paper is illustrated by the MoSo design case, which focused on learning abstract concepts in musical sound. However, the approach could also be applied when designing interactive systems for learning abstract concepts in other fields than music education or even for systems with other purposes than learning (e.g., ). An obvious limitation of our approach is that it is only appropriate when abstract concepts that are to be manipulated are potentially understood metaphorically (when concepts in a target domain are understood in terms of concepts in a source domain ).
11.3 Tangible interaction versus whole-body interaction
In the previously performed Sound Maker study , a whole-bodyinteraction environment was developed. In the present design case, we relied on tangible interaction. When comparing the two prototypes, we see interesting similarities as well as differences. To give an example, in the enactment phase (user study 1) of our MoSo Tangibles design case, we did not find the mapping near–far for pitch. Nevertheless, as a result of the evaluation of our low-fidelity prototypes (user study 2) as well as to enable a comparison to pervious work, we did implement it in MoSo Tangibles. The resulting artifact (the puller) can potentially also be moved via the mapping low–high, simply by rotating the artifact 90 degrees. Although we must conclude from user study 1 that the low–high mapping is dominant over (and should thus make more sense than) near–far, none of the children used the puller in the low–high manner. Apparently, the affordance  of the puller, clearly pointing toward near and far movement, has determined how children use the artifact, rather than their implicit embodied knowledge.
The children interacting with MoSo Tangibles were all able to successfully execute reproduction tasks, whereas some of the participants in the Sound Maker study did not achieve this as the implemented mapping was not discovered within the set timeframe. This difference is obviously due to the different interaction styles, although both styles constrain the interaction possibilities, tangible artifacts allow much clearer and more direct affordances compared to whole-body interaction environments. This particularly holds for the environment used in  in which the only constraint was given by lines on the floor indicating boundaries of the interaction space. One could argue that whole-body interaction environments, such as the one used in the Sound Maker study, therefore encourage relying on embodied knowledge more than tangible systems. On the other hand, if tangible artifacts are correctly designed, they will afford movements based on embodied knowledge, which ensures that children apply embodied knowledge in their understanding of the targeted abstract concepts. The affordances of tangible artifacts may jump start this process, while more discovery is required in whole-body interaction. This shows the importance of successful design of tangible artifacts, which is in our view best achieved through an iterative process in which user involvement plays a major role.
In this paper, we present a people-centered, iterative design approach to the design of interactive learning systems with embodied metaphor-based mappings. In a design case, we have applied this approach to the design of MoSo Tangibles; a tangible system for learning about abstract sound concepts (pitch, volume, and tempo). In this design case, we identified the appropriate embodied metaphors, implemented them in interactive artifacts, and evaluated children’s interactions with MoSo Tangibles. This case revealed that the proposed approach was successful in eliciting and helping us identify an appropriate set of embodied metaphors that children may use in their reasoning about abstract concepts related to sound parameters. Furthermore, verifying the implementation of these metaphors by conducting a second user study ensured effective design of interactive tangibles. The evaluation of MoSo Tangibles has shown that all participants were able to successfully interact with the artifacts after a few minutes of exploration.
Comparing our study to previous work has revealed that although full-body interaction encourages relying on embodied knowledge, tangible systems can provide clarity in interaction by means of affordances and therefore jump start the process of applying specific embodied schemata in reasoning about abstract concepts. This also highlights the importance of successful interaction design, which is in our view best achieved through an iterative and people-centered approach. By proposing and applying such an approach, this paper tries to create a basis for future work on leveraging embodied knowledge in supporting the process of learning abstract concepts.
We thank the children and teachers who participated in our user studies and the music teachers we consulted. Furthermore, we thank our colleagues for their input and help in developing the prototypes as well as setting up and evaluating the (pilot-) user studies. We also thank the reviewers for their constructive comments and advice. In part, this work was funded by the Natural Science and Engineering Research Council of Canada (NSERC).
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
- 1.Processing.org. URL: http://processing.org/. Last accessed 31 March 2010
- 2.Ackermann EK (1996) Perspective-taking and object construction: two keys to learning. In: Constructionism in practice: designing, thinking, and learning in a digital world, pp 25–37Google Scholar
- 3.Ainsworth S (2008) The educational value of multiple-representations when learning complex scientific concepts. In: Visualization: theory and practice in science education, pp 191–208Google Scholar
- 4.Antle AN (2007) The CTI framework: informing the design of tangible systems for children. In: Proceedings of the first international conference on tangible and embedded interaction (TEI’07), pp 195–202Google Scholar
- 5.Antle AN, Corness G, Bakker S, Droumeva M, Hoven E van den, Bevans A (2009) Designing to support reasoned imagination. In: Proceedings of the seventh ACM creativity and cognition conference (C&C’09), pp 275–284Google Scholar
- 6.Antle AN, Droumeva M, Corness G (2008) Playing with the sound maker: do embodied metaphors help children learn? In: Proceedings of the seventh international conference on interaction design and children (IDC’08), pp 178–185Google Scholar
- 7.Bakker S (2009) Move | Learn | Explore. Unpublished Master Thesis, Eindhoven University of Technology, The NetherlandsGoogle Scholar
- 8.Bakker S, Antle AN, Hoven E van den (2009) Identifying embodied metaphors in children’s sound-action mappings. In: Proceedings of the eighth international conference on interaction design and children (IDC’09), pp 140–149Google Scholar
- 9.Bean A, Siddiqi S, Chowdhury A, Whited B, Shaer O, Jacob, RJK (2008) Marble track audio manipulator (MTAM): a tangible user interface for audio composition. In: Proceedings of the second international conference on Tangible and Embedded Enteraction (TEI’08), pp 27–30Google Scholar
- 10.Birchfield D, Ciufo T, Minyard G (2006) SMALLab: a mediated platform for education. In: Proceedings of ACM SIGGRAPH Educators Program, article no. 33Google Scholar
- 11.Bruner JS (1966) Studies in cognitive growth, a collaboration at the center for cognitive studies. John Wiley & Sons, Inc, New YorkGoogle Scholar
- 12.Dourish P (2001) Where the action is: the foundations of embodied interaction. MIT Press, CambridgeGoogle Scholar
- 13.Droumeva M, Antle AN, Wakkary R (2007) Exploring ambient sound techniques in the design of responsive environments for children. In: Proceedings of the first international conference on tangible and embedded interaction (TEI’07), pp 171–178Google Scholar
- 14.Fels S, Gadd A, Mulder A (2002) Mapping transparency through metaphor: towards more expressive musical instruments. Organised Sound 7(2):109–126Google Scholar
- 15.Gibson JJ (1977) The theory of affordances. In: Perceiving, acting, and knowing: toward an ecological psychology. Lawrence Erlbaum, Hillsdale, NJ, USA, pp 67–82Google Scholar
- 16.Girouard A, Solovey ET, Hirshfield LM, Ecott S, Shaer O, Jacob RJK (2007) Smart blocks: a tangible mathematical manipulative. In: Proceedings of the first international conference on tangible and embedded interaction (TEI’07), pp 183–186Google Scholar
- 17.Hansen AS, Overholt D, Burleson W, Jensen CN, Lahey B, Muldner K (2009) Pendaphonics: an engaging tangible pendulum-based sonic interaction experience. In: Proceedings of the eighth international conference on interaction design and children (IDC’09), pp 286–288Google Scholar
- 18.Hashagen A, Büching C, Schelhowe H (2009) Learning abstract concepts through bodily engagement: a comparative, qualitative study. In: Proceedings of the eighth international conference on interaction design and children (IDC’09), pp 234–237Google Scholar
- 19.Horn MS, Jacob RJK (2007) Tangible programming in the classroom with tern. In: Extended abstracts on human factors in computing systems (CHI’07), pp 1965–1970Google Scholar
- 20.Hoven E van den, Frens J, Aliakseyeu D, Martens J, Overbeeke K, Peters P (2007) Design research & tangible interaction. In: Proceedings of the first international conference on Tangible and Embedded Interaction (TEI’07), pp 109–115Google Scholar
- 21.Hummels C, Frens, J (2008) Designing for the unknown: a design process for the future generation of highly interactive systems and products. In: Proceedings of the international conference on engineering and product design education, pp 204–209Google Scholar
- 22.Hurtienne J, Weber K, Blessing L (2008) Prior experience and intuitive use: image schemas in user centred design. In: Designing inclusive futures, pp 107–116Google Scholar
- 23.Hurtienne J, Israel JH (2007) Image schemas and their metaphorical extensions: intuitive patterns for tangible interaction. In: Proceedings of the first international conference on tangible and embedded interaction (TEI’07), pp 127–134Google Scholar
- 24.Jensenius AR (2007) Action—sound: developing methods and tools to study music-related body movement. PhD thesis, University of Oslo, NorwayGoogle Scholar
- 25.Johnson M (1987) The Body in the mind: the bodily basis of meaning, imagination, and reason. Chicago Press, ChicagoGoogle Scholar
- 26.Jordà S, Geiger G, Alonso M, Kaltenbrunner M (2007) The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces. In: Proceedings of the first international conference on tangible and embedded interaction (TEI’07), pp 139–146Google Scholar
- 28.Lakoff G, Johnson M (1980) Metaphors we live by. University of Chicago Press, ChicagoGoogle Scholar
- 29.Marshall P (2007) Do tangible interfaces enhance learning? In: Proceedings of the first international conference on tangible and embedded interaction (TEI’07), pp 163–170Google Scholar
- 30.Marshall P, Price S, Rogers Y (2003) Conceptualising tangibles to support learning. In: Proceedings of the second conference on interaction design and children (IDC’03), pp 101–109Google Scholar
- 31.Mazalek A, Hoven E van den (2009) Framing tangible interaction frameworks. AI EDAM 23(3):225–235Google Scholar
- 32.O’Malley C, Stanton Fraser D (2004) literature review in learning with tangible technologies. URL: http://www.futurelab.org.uk/resources/documents/lit_reviews/Tangible_Review.pdf, last accessed 31 March 2010
- 33.Parreren CFV, Carpay JAM (1972) Sovjetpsychologen aan het woord. Wolters-Noordhoff, GroningenGoogle Scholar
- 35.Resnick M, Martin F, Berg R, Borovoy R, Colella V, Kramer K, Silverman B (1998) Digital manipulatives: new toys to think with. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI’98), pp 281–287Google Scholar
- 36.Schiettecatte B, Vanderdonckt J (2008) AudioCubes: a distributed cube tangible interface based on interaction range for sound design. In: Proceedings of the second international conference on tangible and embedded interaction (TEI’08), pp 3–10Google Scholar
- 37.Svanaes D, Verplank W (2000) In search of metaphors for tangible user intefaces. In: Proceedings of designing augmented reality environments (DARE’00), pp 121–129Google Scholar
- 38.Thompson K (2008) The value of multiple representations for learning about complex systems. In: Proceedings of the eighth international conference for the learning sciences (ICLS’08), vol 2, pp 398–406Google Scholar
- 40.Wessel D (2006) An enactive approach to computer music performance. In: Le Feedback dans la Creation Musical. Studio Gramme, Lyon, France, pp 93–98Google Scholar
- 41.Zigelbaum J, Millner A, Desai B, Ishii H (2006) BodyBeats: whole-body, musical interfaces for children. In: Extended abstracts on human factors in computing systems (CHI’06), pp 1595–1600Google Scholar
- 42.Zuckerman O, Arida S, Resnick M (2005) Extending tangible interfaces for education: digital montessori-inspired manipulatives. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI’05), pp 859–868Google Scholar