1 Introduction

Over the past two decades, an increasing amount of research has explored the use of technology for supporting teaching and learning, recently with more emphasis on multimodal forms of interaction. This has coincided with technological progress and development, where the potential to exploit sensory forms of engagement within learning has expanded. Given increasing evidence from neuroscience and psychophysics on multisensory development and integration, and evidence of the role of an embodied cognition approach to mathematical learning (e.g. [1,2,3,4,5]), the importance of designing and developing multisensory experiences that move beyond the visual to underpin mathematical learning is critical [6]. However, designing and developing effective multisensory solutions is challenging, requiring interdisciplinary cross-team research and development, engaging with practitioners and students. This paper describes the foundational steps towards, and a brief evaluation of the design and development process of one exemplary multisensory learning environment, that demonstrates the role of academic-practitioner partnerships and interdisciplinary research in the development of multimodal interfaces for learning.

1.1 Mathematical comprehension

Literature on mathematical comprehension has focused on several aspects that could influence the ability of children to understand mathematical concepts. Most studies have investigated the impact of learning pathologies related to calculation (i.e. dyscalculia), reading abilities (i.e. dyslexia), or the presence of both [7] on mathematical competence. Also socioeconomic factors seem to influence children’s ability to process mathematical concepts [8]. Neuroscientific studies have focused on the development of cognitive strategies used by children and adults to solve arithmetic problems [9]. In particular, it has been shown that strategies employed by children to solve arithmetic problems change depending on the developmental stage, as confirmed with brain imagining studies [10, 11]. Furthermore, arithmetical and geometrical abilities depend on the target concept to be taught. For instance, many studies identified comprehension of fractions and decimals to be challenging for children and even adults (for a comprehensive review, [11]). The reason behind such difficulties seems to rely on inherent operational difficulties with rational number arithmetic operations. Additionally, prior knowledge on number magnitudes and culturally defined characteristics of the educational system such as the emphasis on the ability to solve mathematical problems seem to be crucial factors in the understanding of fractions and decimals, differences differences of this kind have been observed between Western and East-Asian countries, see [11] for a comprehensive review.

1.2 Multisensory integration and learning difficulties

Learning difficulties might arise from the fact that teaching strategies are typically unisensory. Specifically, considering that most information is conveyed via visual means such as textbooks, whiteboard, etc. (see: [13] for a review), it is plausible to assume that vision is the predominant sense exploited in the classroom. However, humans have multisensory systems to process and interact with their surrounding environment. Studies on multisensory perception have shown how the integration of more sensory modalities improves processing of object properties such as their height [14] or shape [15]. This process is also known as sensory integration and occurs when the brain estimates the same environmental property by processing different sensory modalities [16]. Such integration has often been shown to being optimal by following a Bayesian integration model, that is by integrating unisensory cues based on their reliability thus improving perception of object properties [16]. In other words, redundancy of multisensory stimulation is optimally solved by the brain that merges the sensory cues in a coherent percept based on the level of reliability of the available sensory information. In the context of development, it has been shown that multisensory integration develops quite late during childhood, e.g. after 8 years of life for optimal integration of visual and haptic sensory information [17, 18]. Thus, the development of new technology based on multisensory contingencies could help primary school children to foster their learning abilities [13]. Indeed, neuroscientific results suggest that children might have unisensory preferences about specific concepts to be learned, but they can benefit from multisensory information for their perceptual estimates (e.g. [19]). Along these lines, research on multisensory perception has shown that development of multisensory integration involves several properties of perception. Examples range from the ability to understand object orientation and size via visual or haptic information [17], to the influence of proprioceptive and vestibular information on multisensory integration [18] and spatial navigation [20]. Altogether, these studies suggest on the one hand that multisensory integration is acquired late during the development possibly due to the lack of multisensory experiences at school, and on the other hand that the inclusion of multisensory activities as teaching methods could potentially determine anticipation of multisensory behaviour in children. According to the cross-sensory calibration theory, during primary school age, the strongest sensory modality for specific perceptual properties (e.g. object orientation) calibrates the other senses [21]. From an embodied cognition perspective, research on mathematics education has shown that multisensory teaching approaches play a critical role in the conceptualization of mathematical concepts (e.g. [2, 3, 22]). Moreover, the employment of multisensory perception for educational purposes has been suggested for improving classroom learning [23]. Phenomena such as crossmodal correspondences provide an insightful example of how multisensory perception can be exploited in several contexts. These phenomena consist in multisensory associations between sensory features of different nature, for instance, objects’ size can be identified with auditory stimuli such as sounds of different pitch. Presence of these crossmodal correspondences has been observed in infants [24, 25], children [26], and adults [27, 28]. In the context of more complex operations than size identification, a previous study [29] showed how sounds of different pitch may improve perception of slopes in bar graphs and scatter plots in adults. Altogether, these findings suggest that audition could be used to cue mathematical concepts typically conveyed through visual modes.

1.3 Tangible user interfaces

In the context of video games, and particularly serious games, previous investigation on the state-of-the-art pointed out the importance of the interaction of player-game-context and the efficacy of a given serious game to improve learning [6]. In a 2012 review of education-related video games, the authors highlighted how mathematics-related serious games lacked solid evidence on improved learning [6]. In this context, multisensory-based serious games may provide an insightful and effective tool to foster learning in mathematics [30]. Work on Tangible User Interfaces (TUIs) highlights the importance of binding digital information with actions taken on the real physical world thus enhancing the use of embodied forms of interaction [31]. In the context of growing ubiquitous computing, TUIs augment user’s experience by providing them with a physical extension of the digital environment (e.g. digital manipulatives [32]) thus increasing the interactive experience. The use of TUIs has been implemented in several learning applications, ranging from programming skills [33] to storytelling [34], science [35] as well as in the context of developmental disabilities [36], dyslexia [37], and visual impairment [38]. TUIs have also been proposed as optimal tools to enhance spatial learning in children, likely by improving mental rotation abilities [39]. The effectiveness of TUIs is grounded on the ‘embodied’ properties of such technological solutions, for example, studies show that gesturing about rotating objects improves mental rotation performance in children [40]. Similarly, gestures help children in learning to count by increasing their understanding of one-to-one correspondences [41]. In the context of TUIs, action, and gesture have been shown to improve collaborative problem solving abilities in groups of participants interacting with a tangible tabletop and physical objects [35, 42].

Within the context of mathematics learning, the combination of physical manipulatives with digitally provided symbolic information of TUIs has been argued to support children in making links between symbolic representation in mathematics and their physical correspondence [43]. Properties of the TUIs such as the form of tangible objects are often dictated by the specific learning subjects [31]. Interestingly, a recent and comprehensive review on TUIs [31], reported only a few TUIs on mathematics take advantage of manipulatives. For instance, a tangible-based mixed-reality set-up with a small tabletop projector and depth camera has been tested with children to provide the basis for the development of a TUI aimed at improving non-symbolic math training [44]. Other work explored the feasibility of an embodied experience of mathematics by combining mathematically designed tangibles as a sound design device with an electronic music sequencer as the output device [45]. Engaging aspects of mathematical learning can also be enhanced by the use of a mixed-reality system with tangible interaction that take advantage of the use of a monitor, magnets, and sounds [46] and by the inclusion of more embodied activities, such as physical movement combined with tangible controls and computer visualization [47]. Interestingly, investigation on the use of touch vs. gesture-based interactive devices reported touch to aid performance in mathematic gameplay [48], though the combination of the two has rarely been implemented for mathematical learning. To overcome this issue, force-feedback devices may represent a useful and effective appendix to TUIs. Such devices target the proprioceptive system not only by allowing the user to touch a virtual object but also to provide a gesture-related action input to the digital environment [49] thus providing the means for a functional and interactive user experience that may enhance the correspondence between symbolic and non-symbolic representations.

1.4 Haptic force-feedback

In the last decades, force-feedback devices have been used in much research and applied contexts. Such haptic technology allowed researchers to study the properties of multisensory integration showing optimal combination of visual and haptic information [14]. Recent review articles [49, 50], provided overviews of several applications of haptic technology in the educational context, particularly by highlighting the importance of classifying them based on the area of interest (e.g. medicine, mathematics, physics) and the target educational phase (from elementary to higher and vocational education). The use of force feedback devices indeed spans over applied contexts from virtual medical training for manual intervention [51] and surgery [52] to games [53] and entertainment [54]. In the educational context, the application of force feedback devices is efficient in improving the understanding of key concepts in Physics by simulating applications of hydraulics [55] or friction [56]. Applications for Science teaching (e.g. Newtonian Laws, Space Phenomena, and Mechanics Assembly) indicate that the use of haptic technology increases the immersive feeling thus improving the understanding of such concepts [57]. At an elementary education level, the application of force-feedback for learning Mathematics has been proposed to study abstract operations, such as the negative addition [58]. In detail, the negative addition was represented by visual objects added or removed from a container and by force-feedback that provided the perceived weight changes in the container [58]. In fact, the presence of haptic force-feedback enhances the matching between sensory modalities by compensating the limit of one from the other. In a visual-only abstract representation of the negative addition, haptic information would have been absent thus neglected by the user; instead, the presentation of both sensory cues improves learning and understanding by providing the user with a more vivid and immersive interaction with the simulated environment.

1.5 Present work

This paper draws on evidence that multisensory technology can provide opportunities for enriching the teaching of mathematical concepts from a young age by exploiting other sensory modalities beyond vision [13], to design and develop multisensory learning environments to support young children’s mathematical teaching and learning. In particular, the employment of haptic technology would integrate an efficient additional sensory feedback to the understanding of mathematical concepts. Considering that the application of force feedback devices has been proven to be efficient in the educational context [50], we expect that such a multimodal approach to the teaching of mathematical concepts will reveal itself adaptable to the needs of both teachers and students. In particular, one of the highlighted issues to be solved in the employment of virtual technology is the identification of appropriate learning content [50]. To solve this issue and provide an efficient and tested approach to develop and design a multimodal serious game, in this paper we start from the needs of teachers and students to define the design process based on user-based target concepts and prototypes. The research process took an interdisciplinary approach that: (a) involved teachers from the beginning of the project to identify everyday challenging mathematical concepts and understand which areas of study could potentially benefit from multisensory activities; (b) engaged in a collaborative design process iteratively informing the technical development; and (c) explored ways in which the multisensory features of the developed learning environment might support learning. For the purposes of this paper we take one multisensory environment employing haptic technology to exemplify how this process was executed and delivered into a workable prototype and trialled in the classroom. From this process we draw out key learning design guidelines.

2 Methods

2.1 Research design

This work took a Design Based Research (DBR) approach [59] bringing together educational practitioners with an interdisciplinary research team from Italy and the UK to inform the design, development, and user evaluation of multisensory environments for supporting primary school mathematical learning. DBR takes an inductive approach and is particularly suitable for this task, and educational research contexts, as it aims to impact practice, offer design interventions, and develop theories through the process of design [60]. In this approach “researchers manage research processes in collaboration with participants, design and implement interventions systematically to refine and improve initial designs, and ultimately seek to advance both pragmatic and theoretical aims affecting practice” [61], p. 6. User evaluation took the form of case studies to provide in-depth, in-context examination of usability, descriptions of use in context, and exploration of design opportunities.

The starting point for the design of learning technologies strongly benefits from teachers’ perspectives, as the teacher is a key actor in the use of digital technologies for teaching and learning, offering useful insights from the classroom practice (see guidelines in, [13]). Taking this approach, we aimed to better understand the most challenging mathematical concepts for everyday primary level teaching and learning and to identify the most appropriate mathematical concepts that would benefit from digital multi-sensory activities across Italy and the UK, as well as understanding the feasibility of using sensory modalities beyond vision to teach arithmetic and geometry. We (a) developed and administered a survey questionnaire where teachers were asked to rate how challenging specific arithmetical and geometrical concepts are for primary school children (6–11 y.o.), and (b) ran a number of small participatory design workshops with teachers.

2.2 Teacher survey questionnaires

The purposely designed questionnaire asked teachers to quantify, according to their own experience, the level of difficulty experienced by primary school children (6–11 y.o.) when learning arithmetic and geometric concepts. The questionnaire was anonymous and administrated to 101 Italian primary school teachers recruited through three workshops held in three different Italian cities (Genoa, Milan, and Catania). Arithmetical and geometrical concepts were listed according to the school curricula related to the Italian school system that includes five levels in primary school education: level 1 (6–7 y.o.); level 2 (7–8 y.o.); level 3 (8–9 y.o.); level 4 (9–10 y.o.); level 5 (10–11 y.o.), and which correspond to UK primary key stages 1 and 2.

The questionnaire was divided into four sections. The first two sections aimed to identify difficult concepts in arithmetic (Sect. 1) and geometry (Sect. 2). To classify teachers’ choices, for each education level we asked them to classify the learning difficulty for each concept on the list (see Table 1) as: low, medium, and high. In Sect. 3, we selected five core concepts for arithmetic from Italian and UK school curricula (percentages, equivalences, mental calculation, composition/ordering, size comparison) and four core concepts for geometry (isometric transformations, angles, the Cartesian plane, size comparison. We asked teachers to rate the general learning difficulty for each concept by using the same classification as the previous sections. This section aimed to gain a more general perspective on the learning difficulties in case teachers did not fill in Sects. 1 and 2. Additionally, we included a section that explored the potential use of different sensory modalities to teach specific arithmetic and geometric concepts. Section 4 asked teachers to indicate the sensory modality that, according to them, could be best used to support teaching of specific mathematical concepts (same as those selected for Sect. 3), regardless of what they had experienced to date. Teachers were asked to indicate the potential contribution of three selected sensory modalities—visual, auditory, and haptic—that they thought could be support learning of specific arithmetical and geometrical concepts (see Tables 2, 3). Since the haptic modality involves the use of hands to explore an object or a surface, it was made clear to the teachers that this sensory modality includes the use of the body in interaction with an object (real or simulated), thus including the concept of body movement and proprioception. Given that it is plausible to assume that many teaching strategies rely on visual information, we expect vision to overcome the involvement of other sensory modalities. In this sense, this section of the questionnaire aimed at verifying whether teachers would agree with the use of other sensory modalities than vision in teaching the selected concepts. Given previous findings on the use of auditory and haptic modalities to convey information that is mostly encoded visually [62], we focused on these two sensory modalities. Based on the literature on multisensory integration (e.g. crossmodal correspondences, see Sect. 1.2) the use of such sensory modalities could provide effective cues to support teaching strategies in the classroom.

Table 1 List of arithmetical and geometrical concepts
Full size table
Table 2 Table on the evaluation of multisensory contribution for teaching arithmetical concepts
Full size table
Table 3 Table on the evaluation of multisensory contribution for teaching geometrical concepts
Full size table

2.3 Teacher participant design workshops

For effective design, it is critical to build on current teaching practice, focusing on the particular conceptual challenges identified, fostering real-world experience wherever possible, and to consider cross-concept aspects. Teacher workshops provided this critical link between technical developers, the environment design, and ‘on the ground’ teaching practice. Four workshops were undertaken with teachers in the UK and Italy, with a total of 27 teachers, together with members of the research team. The majority of teachers taught children within the 6–10 y.o. age range, with one teaching children aged 12 y.o. The workshops were structured to invite accounts of teachers’ real-life practice challenges, followed by the generation of possible technical solutions that could be developed. Specifically, they were designed to identify why the concepts noted as challenging from the questionnaires were considered difficult and how a digital multisensory approach might offer different routes into teaching and learning. As such, the workshops served as a foundation for the design of the digital learning experiences drawing on teacher-generated ideas for appropriate activities to support learning of the concepts.

The workshops began with an introduction of the team and the aims and objectives of the project. Teachers then worked in small groups to promote discussion, with a total of 8 groups across all workshops to: verify and note the specific challenging aspects of the mathematical concepts; identify the way these topics are currently taught in school; and explore multisensory solution ideas in conjunction with the project team to inform the underpinning of possible digital designs. This was facilitated through collaborative brainstorming with teachers alongside technical developers, psychologists, and learning scientists to generate the seeds for design ideas. Ideas were explored through small and large group discussions, drawings, and role play.

Conceptual ideas about design and consequent prototypes were trialled in classrooms in the UK and Italy to iteratively inform the learning environment design (see Sects. 3.2 and 4.3).

2.4 Ethics statement

This study was carried out in accordance with the recommendations of the local health service (Comitato Etico ASL 3, Genova, Italy and UCL ethics committee, London, UK, Application 949 number 3923/003). All participants or their legal representatives gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the Comitato Etico ASL 3, Genova, Italy and UCL ethics committee, London, UK, Application 949 number 3923/003.

3 Results

3.1 Results from the questionnaire

Results from Sect. 1 of the questionnaire are represented in Figs. 1 (arithmetic) and 2 (geometry). Regarding arithmetical concepts, most teachers reported challenging concepts to be number representations, number relationships, making sets, predictions and relationships of major, minimum and equal to be challenging at school level 1. From Fig. 1, it can be observed that the concept of percentage only emerged as medium and high difficult at school level 5 (10–11 y.o.) whilst equivalences emerged at levels 3–5 (8–11 y.o.) with the highest difficulties indicated at levels 2–4 (7–10 y.o.). Mental calculation was shown as presenting the biggest challenge starting from level 2 (7–8 y.o.) to school level 4 (9–10 y.o.). Concepts of number relations and representations presented a specific challenge to the youngest students, at level 1. The remaining concepts were not reported to be challenging or reports were low (< 2 teachers). Regarding geometrical concepts, understanding of shapes and spatial indicators were reported as challenging for level 1 students (6–7 y.o.). Orientation (i.e. coordinates) was reported to be particularly challenging for school level 1 and 2 students (6–8 y.o.). Translation and rotations and Cartesian plane concepts were reported to be challenging for older children, at levels 4–5 (9–11 y.o.). Angles were indicated as medium and high difficulty for students at levels 3–4 (8–10 y.o.).

Fig. 1
figure 1

Distribution of teachers’ responses for arithmetics. Each row represents the distribution for the primary school class level (1–5). On the right-hand side, the colour shading bars represent the legends for the response distribution for each subplot. The number of teachers varies for each class level. The concepts’ names reported on the x-axis are taken from the abbreviations in Table 1

Full size image
Fig. 2
figure 2

Distribution of teachers’ responses for geometry. Each row represents the distribution for the primary school class level (1–5). On the right-hand side, the colour shading bars represent the legends for the response distribution for each subplot. The number of teachers varies for each class level. The concepts’ names reported on the x-axis are taken from the abbreviations in Table 1

Full size image

Overall judgments in Sect. 3 are shown in Fig. 3. It can be observed that the arithmetical concept of percentage is reported by more than 50% of teachers as high difficulty whilst mental calculation was identified by the majority of teachers (> 50%) as presenting the medium difficulty for children to learn. Regarding geometric concepts, teachers identified isometric transformations as the concept presenting the highest difficulty (> 50% of teachers); the Cartesian plane and size comparison was reported by the majority of teachers (> 50%) as presenting medium difficulty for children to learn (Fig. 3).

Fig. 3
figure 3

Sum of responses relative to arithmetic (A) and geometric (B) responses

Full size image

The results for Sect. 4 are represented in Fig. 4. Regarding both arithmetic and geometric concepts, teachers indicated vision to be the most useful modality for all concepts (> 50%). Teachers suggested that in their opinion haptic perception could be useful (~ 35%) for arithmetical concepts such as “Very large natural numbers” and “Percentage” while auditory perception was believed to be the least applicable (~ 25%). Regarding geometrical concepts, teachers indicated haptic to be more useful than auditory for teaching “Cartesian planes” (> 35% for haptic) whereas the trend for the other concepts seems to be equally distributed between auditory and haptic modalities (20–25% for both). Although multisensory predictions are overall similar across the investigated concepts, results are shown for each concept with the purpose of providing a full report of the teachers’ survey.

Fig. 4
figure 4

Multisensory predictions. Aggregated responses for teachers’ predictions of the sensory modalities that may be successful in teaching a specific concept. The first row represents responses for arithmetic, the second row represents responses for geometry

Full size image

Isometric transformations and the Cartesian plane were the geometric concepts that teachers identified as presenting the highest difficulty for children to learn. Generally, we have shown that the concepts that are challenging are not consistent across the levels or ages of children. This is partly because some concepts are not introduced to children until a certain age, and partly because the complexity of a concept increases as children progress from level to level (see Sect. 3.1). Some concepts may be easier to understand at a basic level, but become more difficult to understand as the concept is developed and combined with other concepts. Design proposals need to take into account that challenges are different within the age group being targeted.

The administration of the questionnaire allowed us to identify, from the teacher perspective, which arithmetical and geometrical concepts are the most challenging for primary school children. These include arithmetic concepts as percentages, equivalences and mental calculation and geometric concepts as isometric transformations, angles and the Cartesian plane. Generally, the challenging concepts and the associated difficulty level are not consistent across all primary school levels. A possible explanation for this result is that the investigated concepts are generally introduced at different school levels depending on the national curricula. At the same time, the complexity of a concept and its interaction with other concepts both in the same discipline and across the investigated disciplines (e.g. geometry and arithmetic) increases as the children progress from level to level. However, there are exceptions to this observation, like angles and mental calculation are reported to be challenging across most of the school levels investigated in this study. Overall, findings from this questionnaire can be a useful source for studies aimed at investigating learning difficulties in primary school children but also for designing technological solutions addressing these challenges.

Teacher responses to the questionnaire are treated as a starting point for the identification of those arithmetic and geometrical concepts that may benefit from a different teaching approach. On the one hand, we disclosed those concepts that show more learning challenges thus providing a useful hint to pedagogical and psychological investigations of arithmetical and geometrical concepts. On the other hand, this outcome suggests that the actual teaching methodologies might benefit from novel approaches that use other sensory modalities than vision in engaging with arithmetical and geometrical concepts. In this study, the results of the last two sections of the questionnaire indicate that primary school teachers see audition and touch as sensory modalities that can be used to mediate learning of arithmetic and geometry thus supporting the idea that other senses beyond vision might be useful during primary school years, in particular, the auditory modality is not common in mathematics learning. Nevertheless, this pattern of results might depend on the experience of teachers in the classroom. It is plausible to assume that most classroom teaching strategies adopted exploit visual and haptic manipulation rather than auditory stimulation. Since the use of multisensory cues is known to improve perception of an object’s features such as size and shape (see Sect. 1.2), we expect that the integration of other sensory modalities may be useful and effective. Although auditory stimulation is not intuitively related to mathematical concepts, phenomena such as crossmodal correspondences indicate that audition can be exploited to convey concepts not usually related to such sensory modality, such as size or height. Onto this basis, we developed novel technological solutions that use a multisensory approach to covey the mathematical concept of interest.

In this paper, we concentrate on one example prototype; based on the questionnaire results, we designed a serious game that could strengthen concepts such as understanding 2D–3D transformation, mental rotation and percentages. Since the prediction of potential benefits given by the use of a multisensory approach was confirmed by the teachers, we decided to set the basis of the game on the exploitation of visual and haptic information.

3.2 Results from workshops

Results from the teacher workshops verified the mathematical concepts identified through the questionnaires, determined the specific issues that children faced and noted how they were currently taught (see Table 4 for example). In particular, teachers noted challenges with teaching equivalent fractions, simplification of fractions, portioning and the concept of ‘a whole’. In addition, they noted challenges with symmetry (see Table 4), and in conceptualising the properties of 3D Shapes, including a particular difficulty students encounter with the ‘hidden corner’ in the visualisation of a cube.

Table 4 Table on the workshop results on the concept of Symmetry
Full size table

Further similar workshops explored how a multisensory learning experience could be supported through, for example, constructing physical shapes using paper to demonstrate nets or folding paper to explore symmetry. These topics were thought to lend themselves more naturally to multimodal approaches than others, such as number lines. Colour was recognised as a visual resource, but audio or haptic resources were perceived as less commonly used in the classroom. Some activities described by the teachers, such as manipulating paper into 3D shapes, folding to find lines of symmetry or using a trundle wheel suggest that there may be unrecognised haptic or audio aspects.

Three general design considerations also arose for consideration with the prototype development: (1) the need to provide learners with a different sensory experience; (2) building on everyday physical real-world experience is important, thus bringing in multisensory activities; and (3) classrooms are equipped with many different resources, so new (multisensory) technology needs to be useful and valuable to teachers.

A number of design ideas were generated for different mathematical concepts and for geometric challenges including: engaging with number lines through walking where steps are linked to sound (rising pitch for higher number) or haptic feedback (number of taps linked to the number on the line); using the virtual space or (tangibles) digitally augmented objects to explore shape and transformation or the understanding of shape, for example, with a cube the different faces, different angles and inside corner/ ‘hidden’ corner. Here we provide one exemplary multisensory environment design based on results of the questionnaires and design ideas from the workshops, which aimed to support mental rotation, 2D to 3D transformation and percentages: Space Shapes.

4 Space Shapes game design

4.1 Concepts underpinning design

Related concepts identified by teachers as most challenging for children through the questionnaire, combined with the workshop design ideas, were developed into design scenarios, imagining what a future solution might look like [63]. The design of the environments took an iterative process involving technology companies, educators, digital learning researchers and neuroscientists followed by evaluation trials with teachers and students.

In terms of conceptualising the properties of 3D Shapes teachers specifically mentioned that children find counting the corners of a cube challenging because they tend not to include the corner they cannot see. As a result, they tended to omit the “hidden” corner when asked about the number of corners of a cube. To address this challenge, the research team designed a technological prototype which allowed multisensory exploration of a cube employing haptic/ tactile interactions with the internal and external properties of a cube, integrated into two different visual representations of the shape (i.e. 3D form of the cube and its unfolded 2D shape) to foster mental rotation. Specifically, children used an Omni Phantom haptic device (See Fig. 5A) to manipulate a digital cube: i.e. turn it, move it and connect each face of the cube to its respective square represented on the net (see Fig. 5B) using colour coding. A successful connection resulted in the specific face opening up so that children could see inside the cube and use the haptic device to touch its internal corners to collect small objects (i.e. battery fuels) stored there (See Fig. 5C). The use of an Omni Phantom haptic device allowed us to maintain integration between the above-mentioned actions and the narrative of the computer environment which had the form of a serious game. In this context, the use of a haptic real object though likely effective (see Sect. 4.3), would have reduced the integration of the narrative as well as the potentials of exploring virtual objects, thus potentially lowering children engagement in performing the game. Besides, it does not readily lend itself to providing a differential experience of the ‘hidden corner’ through both opaque and transparent visual representations that the digital version enables. The choice of a force-feedback device relies indeed on its peculiarity in allowing the user to both touch a virtual object and provide direct input to the digital environment (see Sects. 1.3 and 1.4). As confirmed by previous research on haptic technology (see Sect. 1.4), the integration of force-feedback increases the user’s immersive experience thus improving learning and understanding, especially in the conversion from concrete to abstract mathematical concepts. Moreover, the use of the Omni Phantom haptic device was not limited to the manipulation of the digital cube. Being a versatile device that allows a more natural manipulation of virtual objects compared to the mouse (for a review, see: [47]), it could be used in each phase of the game (see Sect. 4.2 for details) to explore the game environment and perform the requested actions.

Fig. 5
figure 5

Omni device and interaction with the cube. In A the Omni Phantom haptic Device is depicted. In B an example of connecting the purple face of the cube to the net is represented. In C the exploration of the cube’s interior is represented

Full size image

Two of the four design considerations derived from the teacher workshops (see Sect. 3.2) were integrated with this prototype. First, the environment supported the exploration of the 3D shape using a combination of two modalities (haptic and visual). Second, the serious game provided links with previous knowledge as all children have been acquainted with a cube. However, not all children were acquainted with the unfolded shape of the cube (i.e. the net; see Fig. 6A, B). The environment offered a dynamic link between the 2 representations (connecting the known with the new) aiming to enrich the conceptualization of the cube properties and the transformation from the 3D to the 2D space as well as mental rotation skills that may improve the ability to perform mental transformation in three-dimensional space such as isometric transformation (see Table 1).

Fig. 6
figure 6

Phases of the Space Shapes game. Screenshots of phases 1, 2, 3 and 4 (see Sect. 4.2) are represented in A–D, respectively

Full size image

4.2 Space Shapes game: first prototype

The Space Shapes game was the first prototype of the design process described in this paper. The narrative behind this game was designed to entertain children and facilitate their engagement in the game and the exploited mathematical concepts. In the narrative, the child is introduced to a space girl named Mia who has landed on the moon and needs the player’s help to return home. To help her, the child is asked to reveal a cube where they find and collect fuel cells and rocket pieces; finally, the child builds Mia’s rocket.

The Space Shapes game is divided into 5 phases to focus the child’s attention on different key aspects of the mathematical challenges: (1) introduction and narrative setting to engage the child in the game, (2) find the hidden corners of a cube to engage the child in the tactile exploration of the cube shape, bringing awareness to the all of the corners of the cube, (3) collect the fuel cells from inside each corner, again bringing awareness to all of the internal corners in the cube, (4) build the rocket, where specific engagement with fractions is encouraged, (5) return home to complete the game narrative. In phase 1, Mia is introduced to the child and her need for help is presented (see Fig. 6A). In phase 2, the child is faced with the hidden corners activity that was developed based on the aforementioned observations on children and the role of visual feedback on the haptic control of the game (see Sect. 3.2). In this phase, children need to discover each of the 8 corners of the cube (see Fig. 6B) by means of the haptic device. Once the child has discovered the full cube, this is shown for few seconds and the game moves to the next phase.

In phase 3, the child collects the fuel cells by acting on the cube, not only on its outside areas but also inside (see Fig. 6C). In detail, the child rotates the 3D cube around its centre. This action allows the child to interact with each face of the cube. At this point of the game, the child needs to select the cube’s face and drags the selection to the corresponding unfolded cube represented in the upper left corner of the screen. If the selected face of the unfolded cube corresponds to the dragged cube’s face, the cube opens releasing one piece of the rocket. This procedure aims at reinforcing the ability to convert the 3D figure into the corresponding 2D representation through the direct link between the cube’s face (3D) and the unfolded face (2D) considering their relationship to the other cube’s faces. The child can indeed check the faces on the unfolded cube and understand the neighbouring relationship between cube’s faces on the 3D cube to help their research of the unopened cube’s faces. To collect the fuel cells, the child rotates the cube and haptically explores the interior of the cube where the fuel cells are located. The whole action is then repeated for each cube’s face.

As the collection of rocket pieces and fuel cells advances feedback on the right side of the scene shows the collected fuel expressed in percentages and the number of collected rocket’s pieces both numerically (represented as a fraction, e.g. 2/6) and graphically (represented by highlighting the collected pieces). In phase 4, the child builds Mia’s rocket from the six pieces found and released in the previous phase (see Fig. 6D). By means of the haptic device, the child picks up the rocket pieces and piles them up on a target base positioned at the centre of the scene. Depending on their performance, the child gets feedback if the pieces do not fit (i.e. Mia is sad), or if they do (i.e. Mia cheers up). In this phase, children need to collect each rocket piece located in the front of the scene and, by dragging the piece and moving through the scene, the child needs to release it right on the target base positioned further in-depth compared to the original location of the rocket’s piece. In the middle of this phase, the child is asked to answer mathematical questions. For instance, once 1/3 of the rocket has been built, the child is asked to indicate whether they built 1/2 or 1/3 of the rocket (see Fig. 6D).

Particularly in phases 3 and 4, the haptic exploration provides fundamental multisensory enhancements of the ability to perceive shapes. Although multisensory integration is far from being reached (see Sect. 1.2), already in preschoolers the ability to categorize shapes can improve after experiencing the combination of visual and haptic information coding shapes such as triangles and rectangles [64]. Thus, it follows that the usage of combined visual and haptic information could provide benefits especially in an activity that involves complex manipulations of graphical objects. Considering that in children younger than 8 y.o. size estimation is coded better haptically than visually [17], the use of haptic information would be strongly beneficial in the understanding of such object features. At the same time, older children would exploit the already gained visuo-haptic integration while playing the game. On the other hand, understanding of cube’s properties changes across age with better performance in mental rotation tasks for children aged 10–11 y.o. compared to younger children [65]. Moreover, as observed during the workshop, performance in such a 3D mental rotation task improves when children are presented with a cube whose faces are differently coloured [65]. In total, these aspects have been integrated in the design of the Space Shapes game to exploit perceptual potentials in children taking into account multisensory-related and cognitive peculiarities defined by the developmental stage.

4.3 User design evaluation of the first prototype

A user evaluation of this first prototype was undertaken with twenty year three students (7–8 years) in an in-situ constructive dyad interaction design [66], to promote children’s discussion, collaboration and interaction. All interaction was video recorded to capture: children’s interaction with the haptic device and the game elements, and their discourse, including researcher discussions with the children before and after the games. Data analysis focused on critical episodes of interaction to capture breakthroughs and break-downs in student experience, and user experience with holding and manipulating the haptic device. The study revealed that the visual modality prevailed the haptic modality in the exploration of the mathematical concepts [67]. Specifically, children relied on visual representations (i.e. colour coding, visual representation of objects in the internal corners of the shape) to complete the game.

Another key finding was that the digital implementation of the haptic modality did not have an explicit added value i.e. the advantages of using a haptic device to explore the digital cube as opposed to exploring a physical object were not exploited. Considering that integration of haptic feedback in a meaningful way is rather challenging especially given the limited opportunities of haptic technologies for learning, this first prototype offered valuable information. Specifically, the prototype allowed us to experience what it was like to integrate the haptic feedback in the environment and explore how to refine our design by observing the students and asking them to offer ideas about how to capitalize on the haptic interaction. One student suggested being able to move the cube in such a way that she could not see inside but needed to pick up the fuel cells from the internal corners of the cube by relying on the haptic feedback e.g. by following the contour of the cube base (see [67]). These studies allowed us to refine our design considerations, specifically focusing on the following two aspects: (a) the haptic feedback should differently co-exist with the visual feedback as the latter often prevails and (b) the haptic feedback should integrate information related to the concepts we wanted students to explore (i.e. the hidden corner and the properties of the cube).

Guided by these two key considerations the final design iteration of the digital environment started with the cube initially being invisible, requiring students to haptically follow the contour of the cube to make the cube to visible. One corner was provided as starting point with the aim of encouraging them to hypothesise where the adjacent corner might be and to foster tactile engagement with the edge of the cube that connected the corners. When two linked corners were touched then the relevant edge became visible (see Fig. 7A).

Fig. 7
figure 7

Haptic exploration in the Space Shapes game. The cube emerging through haptic exploration. In A only the cube’s vertices are discovered. In B the first face of the cube is discovered

Full size image

4.4 Final design considerations for the prototype

In the refined version of the prototype, there is a coherent interplay between the haptic and visual feedback, which function complementarily. Specifically, the visual feedback appears as a result of actions with the haptic device which are guided by the haptic feedback. The visual representation of the state of the cube (i.e. position of the visible corners and edges) becomes the basis which further guides haptic exploration e.g. how to move the haptic device so as to complete one face of the cube (see Fig. 7B). This interplay forms the basis of the mathematical information integrated in the haptic feedback as students are expected to haptically touch the digital cube using its properties (e.g. parallel sides, right corners etc.). This structuring is a property which cannot be supported with a physical cube outside a digital environment, offering added value for the digital implementation of the haptic exploration. Of course, students can try and find the corners moving the haptic device randomly and sometimes they do so, but only at the beginning of their interaction. Once they have sufficient information which allows them to start conjecturing about the position of the next corner (for a more detailed analysis see [68]) they use the haptic and visual information concurrently.

5 Discussion and concluding remarks

The identification of learning difficulties from a practitioner perspective seems to corroborate neuroscientific results on the properties of cognitive capabilities such as the ability to mentally manipulate objects processed through multisensory stimulation. In the teaching context, strengthening the use of cognitive skills related to arithmetical and geometrical problems may foster learning of such disciplines. Whenever we interact with the surrounding environment objects properties are encoded via all sensory channels available. From a neuroscientific perspective, brain information processing is for instance most of the time constantly supported by sensorimotor information, even if the body does not produce overt movements of any sort. Along these lines, the motor cortex appears to be active when a cognitive task requires the processing of words and sentences that refer to actions and objects implying motor activity [69, 70]. A similar result has been shown for tasks requiring mental imagery [71], that is the ability to mentally manipulate objects regardless of the encoding sensory modality [72]. Such strong interconnection between mental processes and the body convolves in the theories of embodied cognition, which argue that these two instances cannot be isolated from each other. Along these lines, behavioural evidence shows that better mental rotation is reached when the objects to be manipulated entail greater body-object interaction [73]. Mental rotation can be defined as a specific form of mental imagery which requires a close inspection, manipulation and rotation of mental images of specific objects [74]. Both sensorimotor [75, 76] and haptic concurrent information processing [77] can affect performance in a mental rotation task. Therefore, the ability to mentally rotate objects is influenced by the body and is not strictly related to a specific sensory modality. Translating such neuroscientific findings in learning contexts, it can be noted that the additional use of sensorimotor information represents an effective tool in teaching, for instance, the employment of embodied concepts and ideas can improve children’s reading comprehension [78, 79]. These pieces of evidence provide strong support for the development of novel teaching methods for relevant mathematical domains. In the questionnaire presented here, teachers indicated that in the geometrical context, mental transformations related concepts (e.g. as “Orientation” and “Flip, rotation, translation”) are considered problematic in primary school. Moreover, teachers indicated that the concept of “Isometric Transformations: tipping, translation and rotation” may benefit from the use of other sensory modalities such as haptic. Such achievement can be reached through the use of technologies that offer opportunities to combine the use of visual, haptic (including proprioceptive) information thus extending learner access routes into the same concept. Recently, it has been shown that mental rotation can be achieved with the use of a haptic tablet [80], indicating that haptic-based technology is a good candidate to deliver mental rotation tasks. In the present research, we presented the Space Shapes game that combines visual and haptic/proprioceptive information to solve a mental rotation task that is essential to complete the game. In the second (find the hidden corners of a cube) and third phase of the game (collect the fuel cells), the design aims at enhancing the ability to mentally rotate the cube also in the youngest child by highlighting the different faces of the cube with highly discriminable colours [65]. At the same time, the combination of such visual properties of the cube with the haptic/proprioceptive interaction through the Omni device serves to support and foster the mental representation of the 3D cube through a feedback interaction with the game. The player indeed needs to rotate the cube and match the corresponding face with the unfolded cube to collect the fuel cells, once this action is correctly performed the game can proceed. In particular, the haptic/proprioceptive manipulation renders the interactive process quicker and the simulation closer to real-world interaction with a solid cube. These design choices are thought to enhance the strict interconnection between mental rotation skills and body-related information via coherent multisensory integration of visual and haptic/proprioceptive properties of the gaming action.

In this paper, we also demonstrated how the findings from the teacher questionnaires and workshops were integrated and transformed for the design of a digital learning experience which focused on the exploration of the cube employing haptic and visual representations. Our analysis focusing also on the design of the digital prototype supporting the exploration of the cube, demonstrated that it is not only important to employ different senses in the teaching and learning of mathematical concepts but also to identify the order (which comes first and which follows) when more than one sense is involved, and how they complement each other. Altogether the corroboration of scientific findings on multisensory perception and teachers’ reports point towards fruitful interconnections between disciplines to develop novel technological solutions.

In this study, we followed an approach that started with teacher perspectives, integrated with research evidence to achieve the development of technological solutions based on a multidisciplinary background ranging from neuroscience to pedagogy through technological and engineering development (e.g. [13, 68, 81]). This approach has been followed across the weDRAW project, an EU-H2020-ICT-funded project focusing on multisensory technologies for teaching arithmetic and geometry to primary school children.