It is well known that primary school children may face difficulties in acquiring mathematical competence, possibly because teaching is generally based on formal lessons with little opportunity to exploit more multisensory-based activities within the classroom. To overcome such difficulties, we report here the exemplary design of a novel multisensory learning environment for teaching mathematical concepts based on meaningful inputs from elementary school teachers. First, we developed and administered a questionnaire to 101 teachers asking them to rate based on their experience the learning difficulty for specific arithmetical and geometrical concepts encountered by elementary school children. Additionally, the questionnaire investigated the feasibility to use multisensory information to teach mathematical concepts. Results show that challenging concepts differ depending on children school level, thus providing a guidance to improve teaching strategies and the design of new and emerging learning technologies accordingly. Second, we obtained specific and practical design inputs with workshops involving elementary school teachers and children. Altogether, these findings are used to inform the design of emerging multimodal technological applications, that take advantage not only of vision but also of other sensory modalities. In the present work, we describe in detail one exemplary multisensory environment design based on the questionnaire results and design ideas from the workshops: the Space Shapes game, which exploits visual and haptic/proprioceptive sensory information to support mental rotation, 2D–3D transformation and percentages. Corroborating research evidence in neuroscience and pedagogy, our work presents a functional approach to develop novel multimodal user interfaces to improve education in the classroom.
Similar content being viewed by others
Over the past two decades, an increasing amount of research has explored the use of technology for supporting teaching and learning, recently with more emphasis on multimodal forms of interaction. This has coincided with technological progress and development, where the potential to exploit sensory forms of engagement within learning has expanded. Given increasing evidence from neuroscience and psychophysics on multisensory development and integration, and evidence of the role of an embodied cognition approach to mathematical learning (e.g. [1,2,3,4,5]), the importance of designing and developing multisensory experiences that move beyond the visual to underpin mathematical learning is critical . However, designing and developing effective multisensory solutions is challenging, requiring interdisciplinary cross-team research and development, engaging with practitioners and students. This paper describes the foundational steps towards, and a brief evaluation of the design and development process of one exemplary multisensory learning environment, that demonstrates the role of academic-practitioner partnerships and interdisciplinary research in the development of multimodal interfaces for learning.
1.1 Mathematical comprehension
Literature on mathematical comprehension has focused on several aspects that could influence the ability of children to understand mathematical concepts. Most studies have investigated the impact of learning pathologies related to calculation (i.e. dyscalculia), reading abilities (i.e. dyslexia), or the presence of both  on mathematical competence. Also socioeconomic factors seem to influence children’s ability to process mathematical concepts . Neuroscientific studies have focused on the development of cognitive strategies used by children and adults to solve arithmetic problems . In particular, it has been shown that strategies employed by children to solve arithmetic problems change depending on the developmental stage, as confirmed with brain imagining studies [10, 11]. Furthermore, arithmetical and geometrical abilities depend on the target concept to be taught. For instance, many studies identified comprehension of fractions and decimals to be challenging for children and even adults (for a comprehensive review, ). The reason behind such difficulties seems to rely on inherent operational difficulties with rational number arithmetic operations. Additionally, prior knowledge on number magnitudes and culturally defined characteristics of the educational system such as the emphasis on the ability to solve mathematical problems seem to be crucial factors in the understanding of fractions and decimals, differences differences of this kind have been observed between Western and East-Asian countries, see  for a comprehensive review.
1.2 Multisensory integration and learning difficulties
Learning difficulties might arise from the fact that teaching strategies are typically unisensory. Specifically, considering that most information is conveyed via visual means such as textbooks, whiteboard, etc. (see:  for a review), it is plausible to assume that vision is the predominant sense exploited in the classroom. However, humans have multisensory systems to process and interact with their surrounding environment. Studies on multisensory perception have shown how the integration of more sensory modalities improves processing of object properties such as their height  or shape . This process is also known as sensory integration and occurs when the brain estimates the same environmental property by processing different sensory modalities . Such integration has often been shown to being optimal by following a Bayesian integration model, that is by integrating unisensory cues based on their reliability thus improving perception of object properties . In other words, redundancy of multisensory stimulation is optimally solved by the brain that merges the sensory cues in a coherent percept based on the level of reliability of the available sensory information. In the context of development, it has been shown that multisensory integration develops quite late during childhood, e.g. after 8 years of life for optimal integration of visual and haptic sensory information [17, 18]. Thus, the development of new technology based on multisensory contingencies could help primary school children to foster their learning abilities . Indeed, neuroscientific results suggest that children might have unisensory preferences about specific concepts to be learned, but they can benefit from multisensory information for their perceptual estimates (e.g. ). Along these lines, research on multisensory perception has shown that development of multisensory integration involves several properties of perception. Examples range from the ability to understand object orientation and size via visual or haptic information , to the influence of proprioceptive and vestibular information on multisensory integration  and spatial navigation . Altogether, these studies suggest on the one hand that multisensory integration is acquired late during the development possibly due to the lack of multisensory experiences at school, and on the other hand that the inclusion of multisensory activities as teaching methods could potentially determine anticipation of multisensory behaviour in children. According to the cross-sensory calibration theory, during primary school age, the strongest sensory modality for specific perceptual properties (e.g. object orientation) calibrates the other senses . From an embodied cognition perspective, research on mathematics education has shown that multisensory teaching approaches play a critical role in the conceptualization of mathematical concepts (e.g. [2, 3, 22]). Moreover, the employment of multisensory perception for educational purposes has been suggested for improving classroom learning . Phenomena such as crossmodal correspondences provide an insightful example of how multisensory perception can be exploited in several contexts. These phenomena consist in multisensory associations between sensory features of different nature, for instance, objects’ size can be identified with auditory stimuli such as sounds of different pitch. Presence of these crossmodal correspondences has been observed in infants [24, 25], children , and adults [27, 28]. In the context of more complex operations than size identification, a previous study  showed how sounds of different pitch may improve perception of slopes in bar graphs and scatter plots in adults. Altogether, these findings suggest that audition could be used to cue mathematical concepts typically conveyed through visual modes.
1.3 Tangible user interfaces
In the context of video games, and particularly serious games, previous investigation on the state-of-the-art pointed out the importance of the interaction of player-game-context and the efficacy of a given serious game to improve learning . In a 2012 review of education-related video games, the authors highlighted how mathematics-related serious games lacked solid evidence on improved learning . In this context, multisensory-based serious games may provide an insightful and effective tool to foster learning in mathematics . Work on Tangible User Interfaces (TUIs) highlights the importance of binding digital information with actions taken on the real physical world thus enhancing the use of embodied forms of interaction . In the context of growing ubiquitous computing, TUIs augment user’s experience by providing them with a physical extension of the digital environment (e.g. digital manipulatives ) thus increasing the interactive experience. The use of TUIs has been implemented in several learning applications, ranging from programming skills  to storytelling , science  as well as in the context of developmental disabilities , dyslexia , and visual impairment . TUIs have also been proposed as optimal tools to enhance spatial learning in children, likely by improving mental rotation abilities . The effectiveness of TUIs is grounded on the ‘embodied’ properties of such technological solutions, for example, studies show that gesturing about rotating objects improves mental rotation performance in children . Similarly, gestures help children in learning to count by increasing their understanding of one-to-one correspondences . In the context of TUIs, action, and gesture have been shown to improve collaborative problem solving abilities in groups of participants interacting with a tangible tabletop and physical objects [35, 42].
Within the context of mathematics learning, the combination of physical manipulatives with digitally provided symbolic information of TUIs has been argued to support children in making links between symbolic representation in mathematics and their physical correspondence . Properties of the TUIs such as the form of tangible objects are often dictated by the specific learning subjects . Interestingly, a recent and comprehensive review on TUIs , reported only a few TUIs on mathematics take advantage of manipulatives. For instance, a tangible-based mixed-reality set-up with a small tabletop projector and depth camera has been tested with children to provide the basis for the development of a TUI aimed at improving non-symbolic math training . Other work explored the feasibility of an embodied experience of mathematics by combining mathematically designed tangibles as a sound design device with an electronic music sequencer as the output device . Engaging aspects of mathematical learning can also be enhanced by the use of a mixed-reality system with tangible interaction that take advantage of the use of a monitor, magnets, and sounds  and by the inclusion of more embodied activities, such as physical movement combined with tangible controls and computer visualization . Interestingly, investigation on the use of touch vs. gesture-based interactive devices reported touch to aid performance in mathematic gameplay , though the combination of the two has rarely been implemented for mathematical learning. To overcome this issue, force-feedback devices may represent a useful and effective appendix to TUIs. Such devices target the proprioceptive system not only by allowing the user to touch a virtual object but also to provide a gesture-related action input to the digital environment  thus providing the means for a functional and interactive user experience that may enhance the correspondence between symbolic and non-symbolic representations.
1.4 Haptic force-feedback
In the last decades, force-feedback devices have been used in much research and applied contexts. Such haptic technology allowed researchers to study the properties of multisensory integration showing optimal combination of visual and haptic information . Recent review articles [49, 50], provided overviews of several applications of haptic technology in the educational context, particularly by highlighting the importance of classifying them based on the area of interest (e.g. medicine, mathematics, physics) and the target educational phase (from elementary to higher and vocational education). The use of force feedback devices indeed spans over applied contexts from virtual medical training for manual intervention  and surgery  to games  and entertainment . In the educational context, the application of force feedback devices is efficient in improving the understanding of key concepts in Physics by simulating applications of hydraulics  or friction . Applications for Science teaching (e.g. Newtonian Laws, Space Phenomena, and Mechanics Assembly) indicate that the use of haptic technology increases the immersive feeling thus improving the understanding of such concepts . At an elementary education level, the application of force-feedback for learning Mathematics has been proposed to study abstract operations, such as the negative addition . In detail, the negative addition was represented by visual objects added or removed from a container and by force-feedback that provided the perceived weight changes in the container . In fact, the presence of haptic force-feedback enhances the matching between sensory modalities by compensating the limit of one from the other. In a visual-only abstract representation of the negative addition, haptic information would have been absent thus neglected by the user; instead, the presentation of both sensory cues improves learning and understanding by providing the user with a more vivid and immersive interaction with the simulated environment.
1.5 Present work
This paper draws on evidence that multisensory technology can provide opportunities for enriching the teaching of mathematical concepts from a young age by exploiting other sensory modalities beyond vision , to design and develop multisensory learning environments to support young children’s mathematical teaching and learning. In particular, the employment of haptic technology would integrate an efficient additional sensory feedback to the understanding of mathematical concepts. Considering that the application of force feedback devices has been proven to be efficient in the educational context , we expect that such a multimodal approach to the teaching of mathematical concepts will reveal itself adaptable to the needs of both teachers and students. In particular, one of the highlighted issues to be solved in the employment of virtual technology is the identification of appropriate learning content . To solve this issue and provide an efficient and tested approach to develop and design a multimodal serious game, in this paper we start from the needs of teachers and students to define the design process based on user-based target concepts and prototypes. The research process took an interdisciplinary approach that: (a) involved teachers from the beginning of the project to identify everyday challenging mathematical concepts and understand which areas of study could potentially benefit from multisensory activities; (b) engaged in a collaborative design process iteratively informing the technical development; and (c) explored ways in which the multisensory features of the developed learning environment might support learning. For the purposes of this paper we take one multisensory environment employing haptic technology to exemplify how this process was executed and delivered into a workable prototype and trialled in the classroom. From this process we draw out key learning design guidelines.
2.1 Research design
This work took a Design Based Research (DBR) approach  bringing together educational practitioners with an interdisciplinary research team from Italy and the UK to inform the design, development, and user evaluation of multisensory environments for supporting primary school mathematical learning. DBR takes an inductive approach and is particularly suitable for this task, and educational research contexts, as it aims to impact practice, offer design interventions, and develop theories through the process of design . In this approach “researchers manage research processes in collaboration with participants, design and implement interventions systematically to refine and improve initial designs, and ultimately seek to advance both pragmatic and theoretical aims affecting practice” , p. 6. User evaluation took the form of case studies to provide in-depth, in-context examination of usability, descriptions of use in context, and exploration of design opportunities.
The starting point for the design of learning technologies strongly benefits from teachers’ perspectives, as the teacher is a key actor in the use of digital technologies for teaching and learning, offering useful insights from the classroom practice (see guidelines in, ). Taking this approach, we aimed to better understand the most challenging mathematical concepts for everyday primary level teaching and learning and to identify the most appropriate mathematical concepts that would benefit from digital multi-sensory activities across Italy and the UK, as well as understanding the feasibility of using sensory modalities beyond vision to teach arithmetic and geometry. We (a) developed and administered a survey questionnaire where teachers were asked to rate how challenging specific arithmetical and geometrical concepts are for primary school children (6–11 y.o.), and (b) ran a number of small participatory design workshops with teachers.
2.2 Teacher survey questionnaires
The purposely designed questionnaire asked teachers to quantify, according to their own experience, the level of difficulty experienced by primary school children (6–11 y.o.) when learning arithmetic and geometric concepts. The questionnaire was anonymous and administrated to 101 Italian primary school teachers recruited through three workshops held in three different Italian cities (Genoa, Milan, and Catania). Arithmetical and geometrical concepts were listed according to the school curricula related to the Italian school system that includes five levels in primary school education: level 1 (6–7 y.o.); level 2 (7–8 y.o.); level 3 (8–9 y.o.); level 4 (9–10 y.o.); level 5 (10–11 y.o.), and which correspond to UK primary key stages 1 and 2.
The questionnaire was divided into four sections. The first two sections aimed to identify difficult concepts in arithmetic (Sect. 1) and geometry (Sect. 2). To classify teachers’ choices, for each education level we asked them to classify the learning difficulty for each concept on the list (see Table 1) as: low, medium, and high. In Sect. 3, we selected five core concepts for arithmetic from Italian and UK school curricula (percentages, equivalences, mental calculation, composition/ordering, size comparison) and four core concepts for geometry (isometric transformations, angles, the Cartesian plane, size comparison. We asked teachers to rate the general learning difficulty for each concept by using the same classification as the previous sections. This section aimed to gain a more general perspective on the learning difficulties in case teachers did not fill in Sects. 1 and 2. Additionally, we included a section that explored the potential use of different sensory modalities to teach specific arithmetic and geometric concepts. Section 4 asked teachers to indicate the sensory modality that, according to them, could be best used to support teaching of specific mathematical concepts (same as those selected for Sect. 3), regardless of what they had experienced to date. Teachers were asked to indicate the potential contribution of three selected sensory modalities—visual, auditory, and haptic—that they thought could be support learning of specific arithmetical and geometrical concepts (see Tables 2, 3). Since the haptic modality involves the use of hands to explore an object or a surface, it was made clear to the teachers that this sensory modality includes the use of the body in interaction with an object (real or simulated), thus including the concept of body movement and proprioception. Given that it is plausible to assume that many teaching strategies rely on visual information, we expect vision to overcome the involvement of other sensory modalities. In this sense, this section of the questionnaire aimed at verifying whether teachers would agree with the use of other sensory modalities than vision in teaching the selected concepts. Given previous findings on the use of auditory and haptic modalities to convey information that is mostly encoded visually , we focused on these two sensory modalities. Based on the literature on multisensory integration (e.g. crossmodal correspondences, see Sect. 1.2) the use of such sensory modalities could provide effective cues to support teaching strategies in the classroom.
2.3 Teacher participant design workshops
For effective design, it is critical to build on current teaching practice, focusing on the particular conceptual challenges identified, fostering real-world experience wherever possible, and to consider cross-concept aspects. Teacher workshops provided this critical link between technical developers, the environment design, and ‘on the ground’ teaching practice. Four workshops were undertaken with teachers in the UK and Italy, with a total of 27 teachers, together with members of the research team. The majority of teachers taught children within the 6–10 y.o. age range, with one teaching children aged 12 y.o. The workshops were structured to invite accounts of teachers’ real-life practice challenges, followed by the generation of possible technical solutions that could be developed. Specifically, they were designed to identify why the concepts noted as challenging from the questionnaires were considered difficult and how a digital multisensory approach might offer different routes into teaching and learning. As such, the workshops served as a foundation for the design of the digital learning experiences drawing on teacher-generated ideas for appropriate activities to support learning of the concepts.
The workshops began with an introduction of the team and the aims and objectives of the project. Teachers then worked in small groups to promote discussion, with a total of 8 groups across all workshops to: verify and note the specific challenging aspects of the mathematical concepts; identify the way these topics are currently taught in school; and explore multisensory solution ideas in conjunction with the project team to inform the underpinning of possible digital designs. This was facilitated through collaborative brainstorming with teachers alongside technical developers, psychologists, and learning scientists to generate the seeds for design ideas. Ideas were explored through small and large group discussions, drawings, and role play.
2.4 Ethics statement
This study was carried out in accordance with the recommendations of the local health service (Comitato Etico ASL 3, Genova, Italy and UCL ethics committee, London, UK, Application 949 number 3923/003). All participants or their legal representatives gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the Comitato Etico ASL 3, Genova, Italy and UCL ethics committee, London, UK, Application 949 number 3923/003.
3.1 Results from the questionnaire
Results from Sect. 1 of the questionnaire are represented in Figs. 1 (arithmetic) and 2 (geometry). Regarding arithmetical concepts, most teachers reported challenging concepts to be number representations, number relationships, making sets, predictions and relationships of major, minimum and equal to be challenging at school level 1. From Fig. 1, it can be observed that the concept of percentage only emerged as medium and high difficult at school level 5 (10–11 y.o.) whilst equivalences emerged at levels 3–5 (8–11 y.o.) with the highest difficulties indicated at levels 2–4 (7–10 y.o.). Mental calculation was shown as presenting the biggest challenge starting from level 2 (7–8 y.o.) to school level 4 (9–10 y.o.). Concepts of number relations and representations presented a specific challenge to the youngest students, at level 1. The remaining concepts were not reported to be challenging or reports were low (< 2 teachers). Regarding geometrical concepts, understanding of shapes and spatial indicators were reported as challenging for level 1 students (6–7 y.o.). Orientation (i.e. coordinates) was reported to be particularly challenging for school level 1 and 2 students (6–8 y.o.). Translation and rotations and Cartesian plane concepts were reported to be challenging for older children, at levels 4–5 (9–11 y.o.). Angles were indicated as medium and high difficulty for students at levels 3–4 (8–10 y.o.).
Overall judgments in Sect. 3 are shown in Fig. 3. It can be observed that the arithmetical concept of percentage is reported by more than 50% of teachers as high difficulty whilst mental calculation was identified by the majority of teachers (> 50%) as presenting the medium difficulty for children to learn. Regarding geometric concepts, teachers identified isometric transformations as the concept presenting the highest difficulty (> 50% of teachers); the Cartesian plane and size comparison was reported by the majority of teachers (> 50%) as presenting medium difficulty for children to learn (Fig. 3).
The results for Sect. 4 are represented in Fig. 4. Regarding both arithmetic and geometric concepts, teachers indicated vision to be the most useful modality for all concepts (> 50%). Teachers suggested that in their opinion haptic perception could be useful (~ 35%) for arithmetical concepts such as “Very large natural numbers” and “Percentage” while auditory perception was believed to be the least applicable (~ 25%). Regarding geometrical concepts, teachers indicated haptic to be more useful than auditory for teaching “Cartesian planes” (> 35% for haptic) whereas the trend for the other concepts seems to be equally distributed between auditory and haptic modalities (20–25% for both). Although multisensory predictions are overall similar across the investigated concepts, results are shown for each concept with the purpose of providing a full report of the teachers’ survey.
Isometric transformations and the Cartesian plane were the geometric concepts that teachers identified as presenting the highest difficulty for children to learn. Generally, we have shown that the concepts that are challenging are not consistent across the levels or ages of children. This is partly because some concepts are not introduced to children until a certain age, and partly because the complexity of a concept increases as children progress from level to level (see Sect. 3.1). Some concepts may be easier to understand at a basic level, but become more difficult to understand as the concept is developed and combined with other concepts. Design proposals need to take into account that challenges are different within the age group being targeted.
The administration of the questionnaire allowed us to identify, from the teacher perspective, which arithmetical and geometrical concepts are the most challenging for primary school children. These include arithmetic concepts as percentages, equivalences and mental calculation and geometric concepts as isometric transformations, angles and the Cartesian plane. Generally, the challenging concepts and the associated difficulty level are not consistent across all primary school levels. A possible explanation for this result is that the investigated concepts are generally introduced at different school levels depending on the national curricula. At the same time, the complexity of a concept and its interaction with other concepts both in the same discipline and across the investigated disciplines (e.g. geometry and arithmetic) increases as the children progress from level to level. However, there are exceptions to this observation, like angles and mental calculation are reported to be challenging across most of the school levels investigated in this study. Overall, findings from this questionnaire can be a useful source for studies aimed at investigating learning difficulties in primary school children but also for designing technological solutions addressing these challenges.
Teacher responses to the questionnaire are treated as a starting point for the identification of those arithmetic and geometrical concepts that may benefit from a different teaching approach. On the one hand, we disclosed those concepts that show more learning challenges thus providing a useful hint to pedagogical and psychological investigations of arithmetical and geometrical concepts. On the other hand, this outcome suggests that the actual teaching methodologies might benefit from novel approaches that use other sensory modalities than vision in engaging with arithmetical and geometrical concepts. In this study, the results of the last two sections of the questionnaire indicate that primary school teachers see audition and touch as sensory modalities that can be used to mediate learning of arithmetic and geometry thus supporting the idea that other senses beyond vision might be useful during primary school years, in particular, the auditory modality is not common in mathematics learning. Nevertheless, this pattern of results might depend on the experience of teachers in the classroom. It is plausible to assume that most classroom teaching strategies adopted exploit visual and haptic manipulation rather than auditory stimulation. Since the use of multisensory cues is known to improve perception of an object’s features such as size and shape (see Sect. 1.2), we expect that the integration of other sensory modalities may be useful and effective. Although auditory stimulation is not intuitively related to mathematical concepts, phenomena such as crossmodal correspondences indicate that audition can be exploited to convey concepts not usually related to such sensory modality, such as size or height. Onto this basis, we developed novel technological solutions that use a multisensory approach to covey the mathematical concept of interest.
In this paper, we concentrate on one example prototype; based on the questionnaire results, we designed a serious game that could strengthen concepts such as understanding 2D–3D transformation, mental rotation and percentages. Since the prediction of potential benefits given by the use of a multisensory approach was confirmed by the teachers, we decided to set the basis of the game on the exploitation of visual and haptic information.
3.2 Results from workshops
Results from the teacher workshops verified the mathematical concepts identified through the questionnaires, determined the specific issues that children faced and noted how they were currently taught (see Table 4 for example). In particular, teachers noted challenges with teaching equivalent fractions, simplification of fractions, portioning and the concept of ‘a whole’. In addition, they noted challenges with symmetry (see Table 4), and in conceptualising the properties of 3D Shapes, including a particular difficulty students encounter with the ‘hidden corner’ in the visualisation of a cube.
Further similar workshops explored how a multisensory learning experience could be supported through, for example, constructing physical shapes using paper to demonstrate nets or folding paper to explore symmetry. These topics were thought to lend themselves more naturally to multimodal approaches than others, such as number lines. Colour was recognised as a visual resource, but audio or haptic resources were perceived as less commonly used in the classroom. Some activities described by the teachers, such as manipulating paper into 3D shapes, folding to find lines of symmetry or using a trundle wheel suggest that there may be unrecognised haptic or audio aspects.
Three general design considerations also arose for consideration with the prototype development: (1) the need to provide learners with a different sensory experience; (2) building on everyday physical real-world experience is important, thus bringing in multisensory activities; and (3) classrooms are equipped with many different resources, so new (multisensory) technology needs to be useful and valuable to teachers.
A number of design ideas were generated for different mathematical concepts and for geometric challenges including: engaging with number lines through walking where steps are linked to sound (rising pitch for higher number) or haptic feedback (number of taps linked to the number on the line); using the virtual space or (tangibles) digitally augmented objects to explore shape and transformation or the understanding of shape, for example, with a cube the different faces, different angles and inside corner/ ‘hidden’ corner. Here we provide one exemplary multisensory environment design based on results of the questionnaires and design ideas from the workshops, which aimed to support mental rotation, 2D to 3D transformation and percentages: Space Shapes.
4 Space Shapes game design
4.1 Concepts underpinning design
Related concepts identified by teachers as most challenging for children through the questionnaire, combined with the workshop design ideas, were developed into design scenarios, imagining what a future solution might look like . The design of the environments took an iterative process involving technology companies, educators, digital learning researchers and neuroscientists followed by evaluation trials with teachers and students.
In terms of conceptualising the properties of 3D Shapes teachers specifically mentioned that children find counting the corners of a cube challenging because they tend not to include the corner they cannot see. As a result, they tended to omit the “hidden” corner when asked about the number of corners of a cube. To address this challenge, the research team designed a technological prototype which allowed multisensory exploration of a cube employing haptic/ tactile interactions with the internal and external properties of a cube, integrated into two different visual representations of the shape (i.e. 3D form of the cube and its unfolded 2D shape) to foster mental rotation. Specifically, children used an Omni Phantom haptic device (See Fig. 5A) to manipulate a digital cube: i.e. turn it, move it and connect each face of the cube to its respective square represented on the net (see Fig. 5B) using colour coding. A successful connection resulted in the specific face opening up so that children could see inside the cube and use the haptic device to touch its internal corners to collect small objects (i.e. battery fuels) stored there (See Fig. 5C). The use of an Omni Phantom haptic device allowed us to maintain integration between the above-mentioned actions and the narrative of the computer environment which had the form of a serious game. In this context, the use of a haptic real object though likely effective (see Sect. 4.3), would have reduced the integration of the narrative as well as the potentials of exploring virtual objects, thus potentially lowering children engagement in performing the game. Besides, it does not readily lend itself to providing a differential experience of the ‘hidden corner’ through both opaque and transparent visual representations that the digital version enables. The choice of a force-feedback device relies indeed on its peculiarity in allowing the user to both touch a virtual object and provide direct input to the digital environment (see Sects. 1.3 and 1.4). As confirmed by previous research on haptic technology (see Sect. 1.4), the integration of force-feedback increases the user’s immersive experience thus improving learning and understanding, especially in the conversion from concrete to abstract mathematical concepts. Moreover, the use of the Omni Phantom haptic device was not limited to the manipulation of the digital cube. Being a versatile device that allows a more natural manipulation of virtual objects compared to the mouse (for a review, see: ), it could be used in each phase of the game (see Sect. 4.2 for details) to explore the game environment and perform the requested actions.
Two of the four design considerations derived from the teacher workshops (see Sect. 3.2) were integrated with this prototype. First, the environment supported the exploration of the 3D shape using a combination of two modalities (haptic and visual). Second, the serious game provided links with previous knowledge as all children have been acquainted with a cube. However, not all children were acquainted with the unfolded shape of the cube (i.e. the net; see Fig. 6A, B). The environment offered a dynamic link between the 2 representations (connecting the known with the new) aiming to enrich the conceptualization of the cube properties and the transformation from the 3D to the 2D space as well as mental rotation skills that may improve the ability to perform mental transformation in three-dimensional space such as isometric transformation (see Table 1).
4.2 Space Shapes game: first prototype
The Space Shapes game was the first prototype of the design process described in this paper. The narrative behind this game was designed to entertain children and facilitate their engagement in the game and the exploited mathematical concepts. In the narrative, the child is introduced to a space girl named Mia who has landed on the moon and needs the player’s help to return home. To help her, the child is asked to reveal a cube where they find and collect fuel cells and rocket pieces; finally, the child builds Mia’s rocket.
The Space Shapes game is divided into 5 phases to focus the child’s attention on different key aspects of the mathematical challenges: (1) introduction and narrative setting to engage the child in the game, (2) find the hidden corners of a cube to engage the child in the tactile exploration of the cube shape, bringing awareness to the all of the corners of the cube, (3) collect the fuel cells from inside each corner, again bringing awareness to all of the internal corners in the cube, (4) build the rocket, where specific engagement with fractions is encouraged, (5) return home to complete the game narrative. In phase 1, Mia is introduced to the child and her need for help is presented (see Fig. 6A). In phase 2, the child is faced with the hidden corners activity that was developed based on the aforementioned observations on children and the role of visual feedback on the haptic control of the game (see Sect. 3.2). In this phase, children need to discover each of the 8 corners of the cube (see Fig. 6B) by means of the haptic device. Once the child has discovered the full cube, this is shown for few seconds and the game moves to the next phase.
In phase 3, the child collects the fuel cells by acting on the cube, not only on its outside areas but also inside (see Fig. 6C). In detail, the child rotates the 3D cube around its centre. This action allows the child to interact with each face of the cube. At this point of the game, the child needs to select the cube’s face and drags the selection to the corresponding unfolded cube represented in the upper left corner of the screen. If the selected face of the unfolded cube corresponds to the dragged cube’s face, the cube opens releasing one piece of the rocket. This procedure aims at reinforcing the ability to convert the 3D figure into the corresponding 2D representation through the direct link between the cube’s face (3D) and the unfolded face (2D) considering their relationship to the other cube’s faces. The child can indeed check the faces on the unfolded cube and understand the neighbouring relationship between cube’s faces on the 3D cube to help their research of the unopened cube’s faces. To collect the fuel cells, the child rotates the cube and haptically explores the interior of the cube where the fuel cells are located. The whole action is then repeated for each cube’s face.
As the collection of rocket pieces and fuel cells advances feedback on the right side of the scene shows the collected fuel expressed in percentages and the number of collected rocket’s pieces both numerically (represented as a fraction, e.g. 2/6) and graphically (represented by highlighting the collected pieces). In phase 4, the child builds Mia’s rocket from the six pieces found and released in the previous phase (see Fig. 6D). By means of the haptic device, the child picks up the rocket pieces and piles them up on a target base positioned at the centre of the scene. Depending on their performance, the child gets feedback if the pieces do not fit (i.e. Mia is sad), or if they do (i.e. Mia cheers up). In this phase, children need to collect each rocket piece located in the front of the scene and, by dragging the piece and moving through the scene, the child needs to release it right on the target base positioned further in-depth compared to the original location of the rocket’s piece. In the middle of this phase, the child is asked to answer mathematical questions. For instance, once 1/3 of the rocket has been built, the child is asked to indicate whether they built 1/2 or 1/3 of the rocket (see Fig. 6D).
Particularly in phases 3 and 4, the haptic exploration provides fundamental multisensory enhancements of the ability to perceive shapes. Although multisensory integration is far from being reached (see Sect. 1.2), already in preschoolers the ability to categorize shapes can improve after experiencing the combination of visual and haptic information coding shapes such as triangles and rectangles . Thus, it follows that the usage of combined visual and haptic information could provide benefits especially in an activity that involves complex manipulations of graphical objects. Considering that in children younger than 8 y.o. size estimation is coded better haptically than visually , the use of haptic information would be strongly beneficial in the understanding of such object features. At the same time, older children would exploit the already gained visuo-haptic integration while playing the game. On the other hand, understanding of cube’s properties changes across age with better performance in mental rotation tasks for children aged 10–11 y.o. compared to younger children . Moreover, as observed during the workshop, performance in such a 3D mental rotation task improves when children are presented with a cube whose faces are differently coloured . In total, these aspects have been integrated in the design of the Space Shapes game to exploit perceptual potentials in children taking into account multisensory-related and cognitive peculiarities defined by the developmental stage.
4.3 User design evaluation of the first prototype
A user evaluation of this first prototype was undertaken with twenty year three students (7–8 years) in an in-situ constructive dyad interaction design , to promote children’s discussion, collaboration and interaction. All interaction was video recorded to capture: children’s interaction with the haptic device and the game elements, and their discourse, including researcher discussions with the children before and after the games. Data analysis focused on critical episodes of interaction to capture breakthroughs and break-downs in student experience, and user experience with holding and manipulating the haptic device. The study revealed that the visual modality prevailed the haptic modality in the exploration of the mathematical concepts . Specifically, children relied on visual representations (i.e. colour coding, visual representation of objects in the internal corners of the shape) to complete the game.
Another key finding was that the digital implementation of the haptic modality did not have an explicit added value i.e. the advantages of using a haptic device to explore the digital cube as opposed to exploring a physical object were not exploited. Considering that integration of haptic feedback in a meaningful way is rather challenging especially given the limited opportunities of haptic technologies for learning, this first prototype offered valuable information. Specifically, the prototype allowed us to experience what it was like to integrate the haptic feedback in the environment and explore how to refine our design by observing the students and asking them to offer ideas about how to capitalize on the haptic interaction. One student suggested being able to move the cube in such a way that she could not see inside but needed to pick up the fuel cells from the internal corners of the cube by relying on the haptic feedback e.g. by following the contour of the cube base (see ). These studies allowed us to refine our design considerations, specifically focusing on the following two aspects: (a) the haptic feedback should differently co-exist with the visual feedback as the latter often prevails and (b) the haptic feedback should integrate information related to the concepts we wanted students to explore (i.e. the hidden corner and the properties of the cube).
Guided by these two key considerations the final design iteration of the digital environment started with the cube initially being invisible, requiring students to haptically follow the contour of the cube to make the cube to visible. One corner was provided as starting point with the aim of encouraging them to hypothesise where the adjacent corner might be and to foster tactile engagement with the edge of the cube that connected the corners. When two linked corners were touched then the relevant edge became visible (see Fig. 7A).
4.4 Final design considerations for the prototype
In the refined version of the prototype, there is a coherent interplay between the haptic and visual feedback, which function complementarily. Specifically, the visual feedback appears as a result of actions with the haptic device which are guided by the haptic feedback. The visual representation of the state of the cube (i.e. position of the visible corners and edges) becomes the basis which further guides haptic exploration e.g. how to move the haptic device so as to complete one face of the cube (see Fig. 7B). This interplay forms the basis of the mathematical information integrated in the haptic feedback as students are expected to haptically touch the digital cube using its properties (e.g. parallel sides, right corners etc.). This structuring is a property which cannot be supported with a physical cube outside a digital environment, offering added value for the digital implementation of the haptic exploration. Of course, students can try and find the corners moving the haptic device randomly and sometimes they do so, but only at the beginning of their interaction. Once they have sufficient information which allows them to start conjecturing about the position of the next corner (for a more detailed analysis see ) they use the haptic and visual information concurrently.
5 Discussion and concluding remarks
The identification of learning difficulties from a practitioner perspective seems to corroborate neuroscientific results on the properties of cognitive capabilities such as the ability to mentally manipulate objects processed through multisensory stimulation. In the teaching context, strengthening the use of cognitive skills related to arithmetical and geometrical problems may foster learning of such disciplines. Whenever we interact with the surrounding environment objects properties are encoded via all sensory channels available. From a neuroscientific perspective, brain information processing is for instance most of the time constantly supported by sensorimotor information, even if the body does not produce overt movements of any sort. Along these lines, the motor cortex appears to be active when a cognitive task requires the processing of words and sentences that refer to actions and objects implying motor activity [69, 70]. A similar result has been shown for tasks requiring mental imagery , that is the ability to mentally manipulate objects regardless of the encoding sensory modality . Such strong interconnection between mental processes and the body convolves in the theories of embodied cognition, which argue that these two instances cannot be isolated from each other. Along these lines, behavioural evidence shows that better mental rotation is reached when the objects to be manipulated entail greater body-object interaction . Mental rotation can be defined as a specific form of mental imagery which requires a close inspection, manipulation and rotation of mental images of specific objects . Both sensorimotor [75, 76] and haptic concurrent information processing  can affect performance in a mental rotation task. Therefore, the ability to mentally rotate objects is influenced by the body and is not strictly related to a specific sensory modality. Translating such neuroscientific findings in learning contexts, it can be noted that the additional use of sensorimotor information represents an effective tool in teaching, for instance, the employment of embodied concepts and ideas can improve children’s reading comprehension [78, 79]. These pieces of evidence provide strong support for the development of novel teaching methods for relevant mathematical domains. In the questionnaire presented here, teachers indicated that in the geometrical context, mental transformations related concepts (e.g. as “Orientation” and “Flip, rotation, translation”) are considered problematic in primary school. Moreover, teachers indicated that the concept of “Isometric Transformations: tipping, translation and rotation” may benefit from the use of other sensory modalities such as haptic. Such achievement can be reached through the use of technologies that offer opportunities to combine the use of visual, haptic (including proprioceptive) information thus extending learner access routes into the same concept. Recently, it has been shown that mental rotation can be achieved with the use of a haptic tablet , indicating that haptic-based technology is a good candidate to deliver mental rotation tasks. In the present research, we presented the Space Shapes game that combines visual and haptic/proprioceptive information to solve a mental rotation task that is essential to complete the game. In the second (find the hidden corners of a cube) and third phase of the game (collect the fuel cells), the design aims at enhancing the ability to mentally rotate the cube also in the youngest child by highlighting the different faces of the cube with highly discriminable colours . At the same time, the combination of such visual properties of the cube with the haptic/proprioceptive interaction through the Omni device serves to support and foster the mental representation of the 3D cube through a feedback interaction with the game. The player indeed needs to rotate the cube and match the corresponding face with the unfolded cube to collect the fuel cells, once this action is correctly performed the game can proceed. In particular, the haptic/proprioceptive manipulation renders the interactive process quicker and the simulation closer to real-world interaction with a solid cube. These design choices are thought to enhance the strict interconnection between mental rotation skills and body-related information via coherent multisensory integration of visual and haptic/proprioceptive properties of the gaming action.
In this paper, we also demonstrated how the findings from the teacher questionnaires and workshops were integrated and transformed for the design of a digital learning experience which focused on the exploration of the cube employing haptic and visual representations. Our analysis focusing also on the design of the digital prototype supporting the exploration of the cube, demonstrated that it is not only important to employ different senses in the teaching and learning of mathematical concepts but also to identify the order (which comes first and which follows) when more than one sense is involved, and how they complement each other. Altogether the corroboration of scientific findings on multisensory perception and teachers’ reports point towards fruitful interconnections between disciplines to develop novel technological solutions.
In this study, we followed an approach that started with teacher perspectives, integrated with research evidence to achieve the development of technological solutions based on a multidisciplinary background ranging from neuroscience to pedagogy through technological and engineering development (e.g. [13, 68, 81]). This approach has been followed across the weDRAW project, an EU-H2020-ICT-funded project focusing on multisensory technologies for teaching arithmetic and geometry to primary school children.
The datasets generated for this study are available on request to the corresponding author.
Nunemacher J, Lakoff G, Nunez R (2002) Where mathematics comes from: how the embodied mind brings mathematics into being. Am Math Mon. https://doi.org/10.2307/3072449
Alibali MW, Nathan MJ (2012) Embodiment in mathematics teaching and learning: evidence from learners’ and teachers’ gestures. J Learn Sci. https://doi.org/10.1080/10508406.2011.611446
Nemirovsky R, Rasmussen C, Sweeney G, Wawro M (2012) When the classroom floor becomes the complex plane: addition and multiplication as ways of bodily navigation. J Learn Sci. https://doi.org/10.1080/10508406.2011.611445
Ma JY (2017) Multi-party, whole-body interactions in mathematical activity. Cogn Instr. https://doi.org/10.1080/07370008.2017.1282485
Rosenfeld M (2017) Math on the move: engaging students in whole body learning. Heinemann, London
Young MF, Slota S, Cutter AB et al (2012) Our princess is in another castle: a review of trends in serious gaming for education. Rev Educ Res 82:61–89. https://doi.org/10.3102/0034654312436980
Andersson U (2008) Mathematical competencies in children with different types of learning difficulties. J Educ Psychol. https://doi.org/10.1037/0022-06188.8.131.52
Jordan NC, Levine SC (2009) Socioeconomic variation, number competence, and mathematics learning difficulties in young children. Dev Disabil Res Rev 15:60–68. https://doi.org/10.1002/ddrr.46
Anobile G, Stievano P, Burr DC (2013) Visual sustained attention and numerosity sensitivity correlate with math achievement in children. J Exp Child Psychol 116:380–391. https://doi.org/10.1016/j.jecp.2013.06.006
Peters L, De Smedt B (2018) Arithmetic in the developing brain: a review of brain imaging studies. Dev Cogn Neurosci 30:265–279. https://doi.org/10.1016/j.dcn.2017.05.002
Menon V (2011) Developmental cognitive neuroscience of arithmetic: implications for learning and education. ZDM Mathematics Education 42:515–525. https://doi.org/10.1007/s11858-010-0242-0.Developmental
Lortie-Forgues H, Tian J, Siegler RS (2015) Why is learning fraction and decimal arithmetic so difficult? Dev Rev 38:201–221. https://doi.org/10.1016/j.dr.2015.07.008
Volpe G, Gori M (2019) Multisensory interactive technologies for primary education: from science to technology. Front Psychol 10:1076
Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415:429–433. https://doi.org/10.1038/415429a
Helbig HB, Ernst MO (2007) Optimal integration of shape information from vision and touch. Exp Brain Res 179:595–606. https://doi.org/10.1007/s00221-006-0814-y
Ernst MO, Bülthoff HH (2004) Merging the senses into a robust percept. Trends Cogn Sci 8:162–169. https://doi.org/10.1016/j.tics.2004.02.002
Gori M, Del Viva M, Sandini G, Burr DC (2008) Young children do not integrate visual and haptic form information. Curr Biol 18:694–698. https://doi.org/10.1016/j.cub.2008.04.036
Cuturi LF, Gori M (2019) Biases in the visual and haptic subjective vertical reveal the role of proprioceptive/vestibular priors in child development. Front Neurol 9:1–10. https://doi.org/10.3389/fneur.2018.01151
Jordan KE, Baker J (2011) Multisensory information boosts numerical matching abilities in young children. Dev Sci 14:205–213. https://doi.org/10.1111/j.1467-7687.2010.00966.x
Nardini M, Jones P, Bedford R, Braddick O (2008) Development of cue integration in human navigation. Curr Biol 18:689–693. https://doi.org/10.1016/j.cub.2008.04.021
Gori M (2015) Multisensory integration and calibration in children and adults with and without sensory and motor disabilities. Multisens Res 28:71–99. https://doi.org/10.1163/22134808-00002478
Manches A, O’Malley C (2016) The effects of physical manipulatives on children’s numerical strategies. Cogn Instr. https://doi.org/10.1080/07370008.2015.1124882
Shams L, Seitz AR (2008) Benefits of multisensory learning. Trends Cogn Sci 12:411–417. https://doi.org/10.1016/j.tics.2008.07.006
Peña M, Mehler J, Nespor M (2011) The role of audiovisual processing in early conceptual development. Psychol Sci 22:1419–1421. https://doi.org/10.1177/0956797611421791
Fernández-Prieto I, Navarra J, Pons F (2015) How big is this sound? Crossmodal association between pitch and size in infants. Infant Behav Dev. https://doi.org/10.1016/j.infbeh.2014.12.008
Cuturi LF, Tonelli A, Cappagli G, Gori M (2019) Coarse to fine audio-visual size correspondences develop during primary school age. Front Psychol 10:1–10. https://doi.org/10.3389/fpsyg.2019.02068
Tonelli A, Cuturi LF, Gori M (2017) The influence of auditory information on visual size adaptation. Front Neurosci. https://doi.org/10.3389/fnins.2017.00594
Evans KK, Treisman A (2011) Natural cross-modal mappings between visual and auditory features. J Vis. https://doi.org/10.1167/10.1.6
Parrott S, Guzman-Martinez E, Ortega L et al (2015) Direction of auditory pitch-change influences visual search for slope from graphs. Perception. https://doi.org/10.1177/0301006615596904
Volpe G, Gori M (2019) Multisensory interactive technologies for primary education: from science to technology. Front Psychol. https://doi.org/10.3389/fpsyg.2019.01076
Rodić LD, Granić A (2021) Tangible interfaces in early years’ education: a systematic review. Pers Ubiquitous Comput. https://doi.org/10.1007/s00779-021-01556-x
Zuckerman O, Arida S, Resnick M (2005) Extending tangible interfaces for education: digital montessori-inspired manipulatives. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 859–868
Sapounidis T, Stamelos I, Demetriadis S (2016) Tangible user interfaces for programming and education: a new field for innovation and entrepreneurship. In: Innovation and entrepreneurship in education. Emerald Group Publishing Limited, Bingley, pp. 271–295. https://doi.org/10.1108/S2051-229520160000002016
Di Fuccio R, Mastroberti S (2018) Tangible user interfaces for multisensory storytelling at school: a study of acceptability. Qwerty-Open Interdiscip. J Technol Cult Educ 13(1)
Falcão TP, Price S (2011) Interfering and resolving: how tabletop interaction facilitates co-construction of argumentative knowledge. Int J Comput Collab Learn 6:539–559
Falcão TP, Price S (2012) Independent exploration with tangibles for students with intellectual disabilities. In: Proceedings of the 11th international conference on interaction design and children, pp 236–239
Fan M, Antle AN, Cramer ES (2016) Design rationale: opportunities and recommendations for tangible reading systems for children. In: Proceedings of the the 15th international conference on interaction design and children, pp 101–112
Jafri R, Aljuhani AM, Ali SA (2015) A tangible interface-based application for teaching tactual shape perception and spatial awareness sub-concepts to visually impaired children. Proc Manuf 3:5562–5569
Baykal GE, Alaca IV, Yantaç AE, Göksun T (2018) A review on complementary natures of tangible user interfaces (TUIs) and early spatial learning. Int J Child Comput Interact 16:104–113. https://doi.org/10.1016/j.ijcci.2018.01.003
Ping R, Ratliff K, Hickey E, Levine S (2011) Using manual rotation and gesture to improve mental rotation in preschoolers. In: Proceedings of the annual meeting of the cognitive science society
Alibali MW, Dirusso AA (1999) The function of gesture in learning to count: more than keeping track. Cogn Dev 14:37–56. https://doi.org/10.1016/S0885-2014(99)80017-3
Anastasiou D, Maquil V, Ras E (2014) Gesture analysis in a case study with a tangible user interface for collaborative problem solving. J Multimodal User Interfaces 8:305–317. https://doi.org/10.1007/s12193-014-0158-z
Moeller K, Fischer U, Nuerk H-C, Cress U (2015) Computers in mathematics education—training the mental number line. Comput Hum Behav 48:597–607
Salman E, Besevli C, Göksun T et al (2019) Exploring projection based mixed reality with tangibles for nonsymbolic preschool math education. In: Proceedings of the thirteenth international conference on tangible, embedded, and embodied interaction, pp 205–212
Velamazán M (2016) Superbleeper: a tangible system for electronic music & embodied mathematics. Proc Audio Mostly 2016:278–284
Marichal S, Rosales A, Perilli FG et al (2017) Ceta: designing mixed-reality tangible interaction to enhance mathematical learning. In: Proceedings of the 19th international conference on human–computer interaction with mobile devices and services, pp 1–13
Mickelson J, Canton M, Ju W (2011) Pattern poses: embodied geometry with tangibles and computer visualization. In: Proceedings of the 10th international conference on interaction design and children, pp 242–245
Jong J-T, Hong J-C, Yen C-Y (2013) Persistence temperament associated with children playing math games between touch panel and embodied interaction. J Comput Assist Learn 29:569–578
Baud-Bovy G, Balzarotti N (2017) Using force-feedback devices in educational settings: a short review. In: MIE 2017—proceedings of 1st ACM SIGCHI international work multimodal interact education co-located with ICMI 2017. https://doi.org/10.1145/3139513.3139518
Liu LM, Li W, Dai JJ (2017) Haptic technology and its application in education and learning. In: Ubi-media 2017—proceedings of 10th international conference on Ubi-media computing with 4th international work on advantage E-learning 1st international work on multimedia IoT networks, systems and applications.https://doi.org/10.1109/UMEDIA.2017.8074138
Ullrich S, Kuhlen T (2012) Haptic palpation for medical simulation in virtual environments. IEEE Trans Vis Comput Graph 18:617–625
Escobar-Castillejos D, Noguez J, Neri L et al (2016) A review of simulators with haptic devices for medical training. J Med Syst 40:104
Kessler JA, Lovelace RC, Okamura AM (2013) A haptic system for educational games: design and application-specific kinematic optimization. In: Dynamic systems and control conference, p V002T26A006
Carvalho B, Soares M, Neves A et al (2016) Virtual reality devices applied to digital games. A literature review. Ergon Des Methods Tech 125–136
Hamza-lup FG, Adams M (2008) Feel the pressure : E-learning systems with haptic feedback opportunities exist for development of novel educational. In: Symposium on haptic interfaces virtual environment teleoperator system, pp 445–450
Hamza-Lup FG, Stanescu IA (2010) The haptic paradigm in education: challenges and case studies. Internet High Educ 13:78–81. https://doi.org/10.1016/j.iheduc.2009.12.004
Pantelios M, Tsiknas L, Christodoulou S, Papatheodorou T (2004) Haptics technology in educational applications, a case study. J Digit Inf Manag 2:171–178
Singapogu RB, Burg TC (2009) Haptic virtual manipulatives for enhancing K-12 special education. In: Proceedings of the 47th annual southeast regional conference, pp 1–4
Anderson T, Shattuck J (2012) Design-based research: a decade of progress in education research? Educ Res. https://doi.org/10.3102/0013189X11428813
O’leary Z (2017) The essential guide to doing your research project. Sage, London
Wang F, Hannafin MJ (2005) Design-based research and technology-enhanced learning environments. Educ Technol Res Dev 53(4):5–23
Spence C (2011) Crossmodal correspondences: a tutorial review. Atten Percept Psychophys 73:971–995. https://doi.org/10.3758/s13414-010-0073-7
Rosson MB, Carroll JM (2002) Usability engineering: scenario-based development of human–computer interaction
Kalenine S, Pinet L, Gentaz E (2011) The visual and visuo-haptic exploration of geometrical shapes increases their recognition in preschoolers. Int J Behav Dev 35:18–26. https://doi.org/10.1177/0165025410367443
Lütke N, Lange-Küttner C (2015) Keeping it in three dimensions: measuring the development of mental rotation in children with the rotated colour cube test (RCCT). Int J Dev Sci 9:95–114. https://doi.org/10.3233/DEV-14154
Als BS, Jensen JJ, Skov MB (2005) Comparison of think-aloud and constructive interaction in usability testing with children. In: Proceedings of the 2005 conference on interaction design and children. ACM, New York, pp 9–16
Yiannoutsou N, Johnson R, Price S (2018) Exploring how children interact with 3D shapes using haptic technologies. In: IDC 2018—proceedings of the 2018 ACM conference on interaction design and children
Price S, Yiannoutsou N, Johnson R, Outhwaite L (2020) Enacting elementary geometry: participatory ‘haptic’ sense-making. Digit Exp Math Educ. https://doi.org/10.1007/s40751-020-00079-z
Aziz-Zadeh L, Wilson SM, Rizzolatti G, Iacoboni M (2006) Congruent embodied representations for visually presented actions and linguistic phrases describing actions. Curr Biol. https://doi.org/10.1016/j.cub.2006.07.060
Tomasino B, Werner CJ, Weiss PH, Fink GR (2007) Stimulus properties matter more than perspective: an fMRI study of mental imagery and silent reading of action phrases. Neuroimage. https://doi.org/10.1016/j.neuroimage.2007.03.035
Zhang M, Weisser VD, Stilla R et al (2004) Multisensory cortical processing of object shape. Cogn Affect Behav Neurosci 4:251–259
Kosslyn S (2005) Mental images and the brain. Cogn Neuropsychol 22:333–347
Suggate S, Lehmann J, Stoeger H, Jansen P (2019) Cognition embodied: mental rotation is faster for objects that imply a greater body–object interaction. J Cogn Psychol 31:876–890. https://doi.org/10.1080/20445911.2019.1678627
Gill HS, O’Boyle MW, Hathaway J (1998) Cortical distribution of EEG activity for component processes during mental rotation. Cortex. https://doi.org/10.1016/S0010-9452(08)70774-3
Shenton JT, Schwoebel J, Coslett HB (2004) Mental motor imagery and the body schema: evidence for proprioceptive dominance. Neurosci Lett 370:19–24. https://doi.org/10.1016/j.neulet.2004.07.053
Fischer MH, Zwaan RA (2008) Embodied language: a review of the role of the motor system in language comprehension. Q J Exp Psychol 61:825–850. https://doi.org/10.1080/17470210701623605
Lohmann J, Rolke B, Butz MV (2017) In touch with mental rotation: interactions between mental and tactile rotations and motor responses. Exp Brain Res 235:1063–1079. https://doi.org/10.1007/s00221-016-4861-8
Glenberg AM, Goldberg AB, Zhu X (2011) Improving early reading comprehension using embodied CAI. Instr Sci. https://doi.org/10.1007/s11251-009-9096-7
Glenberg AM (2011) How reading comprehension is embodied and why that matters. Int Electron J Elem Educ 4(1):5–18
Tivadar RI, Rouillard T, Chappaz C et al (2019) Mental rotation of digitally-rendered haptic objects. Front Integr Neurosci 13:1–9. https://doi.org/10.3389/fnint.2019.00007
Gori M, Volpe G, Cappagli G, Volta E, Cuturi LF (2021) Embodied multisensory training for learning in primary school children. 2021 IEEE Inter Conf Dev Learn (ICDL). https://doi.org/10.1109/ICDL49984.2021.9515596
This research was part of the weDRAW project (wedraw.eu), which received funding from the European Union’s Horizon 2020 Research and Innovation Programme, Grant Agreement No. 732391. We thank DeAScuola (DeAgostini Scuola) and Vanda Gatti for the support in the questionnaire administration. The digital environment (Space Shapes) discussed here was developed by Neville Rudd, Learn-Tpm. We would like to thank the students and teachers of the UK and Italian schools who participated in the study.
Open access funding provided by Istituto Italiano di Tecnologia within the CRUI-CARE Agreement. This work was partially supported by the EU- H2020-ICT Project WeDRAW. WeDRAW has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement No. 732391.
Conflict of interest
The authors declared that they have no conflict of interest.
Consent to participate
Consent for publication
This study was carried out in accordance with the recommendations of the local health service (Comitato Etico, ASL 3, Genova, Italy and UCL ethics committee, London, UK, Application number 3923/003). All subjects or their legal representatives gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the Comitato Etico, ASL 3, Genova, Italy and the UCL ethics committee, London, UK, Application number 3923/003.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Cuturi, L.F., Cappagli, G., Yiannoutsou, N. et al. Informing the design of a multisensory learning environment for elementary mathematics learning. J Multimodal User Interfaces 16, 155–171 (2022). https://doi.org/10.1007/s12193-021-00382-y