An increasing body of work within the embodied cognition paradigm provides evidence of the importance of sensorimotor experience for cognition and mathematical learning (e.g. Barsalou 2008; Cress et al. 2010; Abrahamson and Trninic 2011). A central tenet is around the role of particular action experiences: (a) in the development of action schemes as foundations for new ways of conceptualising mathematical ideas (Abrahamson and Sánchez-García 2016) and (b) in providing sensorimot orcontingencies that underpin gestural forms of communication that can be used later for explaining mathematical ideas (e.g. Gerofsky 2012; Alibali and Nathan 2012; Gallagher and Lindgren 2015; Johnson-Glenberg 2018).

Sensorimotor contingencies draw on notions that perception through active engagement is intimately linked to skillful action, rather than fully computed in the brain (Buhrmann et al. 2013). The increasing potential for digital learning experiences to be designed to foster specific movement and sensory experience highlights the need to understand better ways in which these environments shape young children’s interaction and cognition. For example, children’s arm movements can be dynamically linked to visualisations that show changes in angle size (Walkington et al. 2014).

Multisensory digital learning environments offer new ways for children to engage with mathematical ideas through multimodal forms of engagement. Multimodal resources provide access to information from different sensory perspectives and opportunities for participatory sense making where active participation is central to developing meaning around objects and interaction (De Jaegher and Di Paolo 2007). Haptic technologiesuse input and output devices, such as data gloves or joysticks, allowing users to feel different sensations through force feedback or vibrating sensors, and are commonly linked to related visual representations.

Such technologies particularly bring the tactile into interaction, supporting constructivist, embodied forms of learning through sensory active learning experiences. Furthermore, they offer opportunities for exploiting perceptual and bodily experiences, by designing environments that shape particular actions and gestures during interaction. As Bivall et al. (2011) claim, “Providing haptic experience of a phenomenon, in addition to its visual aspects, would be expected to radically impact upon how people understand it” (p. 703), which enables learners to build richer multimodal representations (Zacharia 2015).

Our study sought to identify the role of haptic interaction (through mediated action and tactile feedback) in fostering specific action experiences that might underpin conceptualisations of 3D shape, and the role of the body in participatory sense making with children aged 7–8 years. A purposefully designed learning environment dynamically combined haptic experience of 3D shape through tactile interaction in 3D space with a 2D digital visualisation. This brings forward new sensorimotor experiences for children to draw on: for example, tactile sensations as they move along edges, vertices and corners that form the key properties of shape create sensorimotor pathways related to geometrical learning.

By analysing in situ dynamic unfolding of interaction with pairs of children as they engage with mathematical ideas around 3D shape, our study sought to answer the following research questions: how can haptic interaction foster specific action experiences that shape the way that children conceptualise 3D shape? How do these experiences shape children’s communication of ideas around 3D shape? How do children use their body as an instrument for participatory sense making while interacting with the haptic device?

Background

This section draws on relevant ideas from embodied cognition to situate the design of the haptic environment, and the role it might have in supporting cognition, followed by a brief review of research on the use of haptic technologies for education, with a focus on mathematics.

Embodied Cognition

Recent research on embodied cognition shows increasing evidence for the role of the body in learning and development, and how it provides a concrete foundation for more abstract conceptualisation. In the context of the work reported on here, we draw on three key ideas: conceptual metaphor, enactive cognition and gesture as a form of evidence. While distinctions have been made between embodiment as interactionist and embodiment as conceptualist (Stevens 2012), we argue that these are not in tension with one another. The interactionist perspective focuses on the active, participatory, social construction of meaning through dynamic interaction among body (considering its specific biological and physical affordances), environment and others (Varela et al. 1991), while the conceptualist perspective focuses on how this constructed meaning forms the bodily basis for reasoning and abstraction in a wider sense.

Firstly, theories of conceptual metaphor (Lakoff and Johnson 1980) argue for the central role of sensorimotor experiences in forming image-schemas and use of language, which underpin our conceptualisation of experienced phenomena (Hampe 2005). Physical concrete experiences also form metaphorical analogies for abstract ideas. This highlights the role of our physical experience in shaping the way we reason, think and talk about the world, not only concrete ideas, but also abstract ones (Lakoff and Johnson 1980). Experiences of up, down, backwards, forwards, inside and outside shape how we conceptualise positioning in space, as well as underpinning more abstract ideas. For instance, the notion of ‘progress’ might be described as moving forwards or be in a backward trend.

Secondly, enactive cognition proposes that understanding is actively constructed through dynamic interactions among body (considering its specific biological and physical affordances), environment and others (Varela et al. 1991; Gallagher and Lindgren 2015; Towers and Martin 2015). Within this paradigm, the concept of participatory sense-making places emphasis on active participation in developing meaning around objects/interaction through dynamic social coupling (van Dijk 2018):

The softness of a sponge is not to be found ‘in it’ but in how it responds to the active probing and squeezing of our appropriate bodily movements (e.g., with the fingers or the palms of the hand). It is the outcome of a particular kind of encounter between a ‘questioning’ agent with a particular body (sponges are solid ground for ants) and a ‘responding’ segment of the world. (De Jaegher & Di Paolo, 2007, p. 489)

Enacting with a tool or object is also central to the concept of enactive metaphors, metaphors that, “we enact […] or one that we bring into existence through our action” (Gallagher and Lindgren 2015, p. 392): for example, swinging the forearm with a fixed elbow enacts the motion of a pendulum (Lindgren and Johnson-Glenberg 2013). Enactive metaphors are argued to be important in fostering learner transfer of an action from one context to another – a critical feature here being the gesture, movement or action (Winner et al. 1979).

Enactive metaphors have been shown to be valuable for learning, specifically the re-enaction of modality-specific experiences (Barsalou 2008), and where the environment design fosters learners to move “in a prescribed way or play-act a specified process” (Gallagher and Lindgren 2015, p. 398). For example, in MEteor, motion tracking and projected visualisations encouragedlearners to enact the trajectory of an asteroid (Lindgren et al. 2015).

Thirdly, research from various disciplines show the important role of gesture in providing insights into children’s and adults’ thinking and understanding (e.g. Gerofsky 2012; Goldin-Meadow 2011; Callinan 2014), supporting real-time reasoning (e.g. Crowder 1996; McNeill 1992)and enhancing communication (Alibali and Nathan 2012; Thomas Jha et al. under review). Research on gesture suggests the importance of particular underlying action experiences that give rise to the specific gestures that are used to explain, express or externalise (mathematical) ideas:“Gestures are more than hand-waving—and must connect underlying concepts, pedagogical language, and student understanding” (Weisberg and Newcombe 2017, p.2).

Haptic technologies provide opportunities for designing environments that shape particular actions during interaction, exploit perceptual and bodily experiences, and bring together multimodal and multisensorial experiences. They can also generate motor schemas that ground concepts (Zacharia 2015). “Introducing constraints can help learners become perceptually attuned to relevant affordances for performinga specific skill” (Abrahamson and Sánchez-García 2016, p.212). The design of the haptic environment reported on here introduced such constraints, aiming to underpin action schemes for action-based mathematical learning about 3D shape. For example, children had to rely on tactile sensing of the edge of a shape as a guide to tracing its edges, vertices and faces, hence structuring action that supports tactile exploration of the cube.

Collectively, these bodies of work suggest the value of exploring the role of haptic technologies combined with the visual in supporting children’s embodied mathematical experience. Firstly, conceptual metaphor argues for the role of sensorimotor experience in underpinning how we conceptualise the world, evidenced through language. Secondly,the notion of enactive metaphors suggests the importance of specific body movements and action sequences in providing meaningful action experiences that shape thinking and reasoning. Thirdly, gesture research provides evidence for the role of gesture in providing insights into cognition and the role of embodied action in providing sensorimotor representations that underpin it.

Geometry Learning: Shape

Children have been shown to encounter particular challenges with integrating 2D and 3D representations of shape, and characteristics such as the ‘hidden corner’ (Price et al. 2017; Fiantika et al. 2018). Pundkar (2018) showed children 7–8 years old could draw 2D shapes, but failed to draw 3D shapes, and were unable to describe the difference between them. Duval (2005) distinguishes between iconic and non-iconic ways of ‘seeing’ shape: an iconic perspective sees the contours of the shape (whether 2D or 3D) as a whole; whereas non-iconic relates to the ability to break the figure down into components, such as faces, edges, vertices – that also can be transformed into another figure (e.g. a face on a cube becomes a 2D square).

The process of splitting up an object into sub-parts is important in problem solving in geometry (Duval 2005;see Nicolas and Trgalova 2019), and for identifying and reasoning about geometric properties (Laborde 2008). Laborde comments that such problem solving,“can be supported by adding some elements on the diagram and/or hiding other elements” (p.39). In addition to the features of a shape, her piece shows that meaning is also constructed through ways in which learners can use representations.

The environment reported here provides a haptic tool that enables construction of a 3D object guided by haptic and visual feedback, providing progressive construction of a cube through engagement with its sub-parts (namely edges and vertices) and through its manipulation. Drawing on instrumentation theory, Laborde proposes that, in working with new tools, students have to learn how to use the tool and, in so doing, develop knowledge of the domain. In the study reported on here, children were learning how to use the haptic device, which fostered particular action repertoires in relation to construction of 3D shape.

Haptic Technologies for Education

The term ‘haptic’ derives from the Greek terms haptikos, meaning ‘able to touch’, and haptesthai, meaning ‘able to lay hold of’ (Révész 1950). Through force and tactile feedback haptic technologies enable simulated tactile sensations of an object’s hardness, shape and texture: for example, haptic information about molecular forces, in conjunction with a 3D visual protein model, can improve protein–ligand understanding (Bivall et al. 2011). Importantly, haptics involves active touch, in that, “the individual deliberately chooses his or her actions in the exploration and manipulation of an object” (Minogue and Jones 2006, p.332), our sense of touch being instrumental in our ability to modify and manipulate the world around us (McLaughlin et al. 2002).

Thus, while haptic feedback provides tactile experience (e.g. sensation of touching an object), it can also foster specific movements or kinesthetic experience (e.g. where a haptic device is used to trace a line of tactile resistance). In this way, haptic interaction provides a ‘new modality’ of experience (Wiebe et al. 2009) which functions, “at the intersection of dynamic visualisation and physical manipulatives” (p.51). This provides new learning experiences that offer novel ways of engaging and conceptualising ideas in mathematics, and thus the potential for re-shaping children’s thinking and reasoning (Vygotsky, 1930/Vygotsky 1978).

Studies in haptics for education primarily involve undergraduate or high-school participants in science education (e.g. Koul et al. 2013; Minaker et al. 2016), with few addressing mathematics (Sourin and Wei 2009) or elementary learning. The majority of studies take an experimental approach and show mixed results of the benefits of haptic visual manipulatives over basic visual manipulatives, with increased immersion and engagement being one benefit (Jones et al. 2006). Haptic-augmented feedback was found to be beneficial for elementary learning (better recall, inference and transfer), given multimodal experience in a simulation environment for learning about how gears work (Han and Black 2011) and for developing psychomotor skills (Zacharia and Michael 2015). In the context of mathematics, Davis et al. (2017) found that haptics brought new perspectives on the concept being explored, including partial or unfamiliar perspectives of shapes. Their findings suggest that, “learning abstractions goes beyond familiarization or sensorimotor memory” (p.52), but further research is needed to understand better its role in conceptual development.

Sensory experience and mindful movement (e.g. positioning arms to create a 90-degree angle) have been shown to be important in understanding geometric figures (e.g. Hall and Nemirovsky 2012), and learning angles and shape (Shoval 2011; Kaur 2013; Ma 2017), as well as synergies between combined use of different artefacts – digital and physical (Mariotti and Montone 2020). Yet little work has looked at haptic technologies in this space, with the exception of Güçler et al. (2013), who investigated groups of 10-year-old students’ discourse around haptic interaction with 3D objects, providing a (rarer) qualitative approach to examining student interaction in a visuo-haptic environment. Their preliminary findings suggest haptic interaction canfoster meaningful discourse, including exploring, conjecturing, negotiating meaning, and sense-making. Our study extends this work by taking an embodied cognition perspective that engages with the specific affordances of sensing body action or manipulation and links to visual, audio or tactile representations, which have not, to date, been explored in relation to embodied mathematical experience.

SpaceBalloons and SpaceShapes

SpaceBalloons and SpaceShapes were designed to enable 3D exploration of space and shape through physical force feedback experienced through the OmniPhantom haptic device (see Fig. 1) with visual and audio feedback.

Fig. 1
figure 1

Omniphantom device, old generation, fire wire connected

Full size image

SpaceBalloons was designed as a pre-requisite to SpaceShapes in order to familiarise children with using the OmniPhantomthrough supporting exploration of spatial concepts (i.e. depth, construction of a spatial mental map) based on tactile and sound feedback (see Fig. 2). Balloons are situated in an adjustable grid (3 × 3 or 5 × 5), the aim being to use the haptic device to pop all of the balloons. A haptic grid, in the form of tracks can guide action: when the haptic device (visually represented by a purple ball; Fig. 2) is placed on these tracks,users can ‘feel’ the walls of the tracks. Each balloon is attached to a ‘target’ base. When the target is ‘touched’ by the haptic device, force feedback gives a sense of going up or down a step, indicating the base of a balloon. When the haptic device is moved up the string to meet the balloon, the balloon pops, with a sound.

Fig. 2
figure 2

SpaceBalloons: tracks, targets and the haptic device representation as a purple ball

Full size image

SpaceShapes was designed to support 3D exploration of shape specifically, enabling tactile combined with visual exploration of properties of 3D and 2D shape (a cube), with an emphasis on faces, edges and vertices, as well as 2D to 3D transformations. Force feedback resistance provided the sensation of moving through air and then feeling something like a wall when the haptic device made contact with a face of the cube (tactile and kinesthetic). The resistance changed as the stylus moved over the edges and vertices. This provided a tactile sensation of running along a hard edge or over sharp corner points with the stylus, and moving into a void (no resistance) as the stylus moved away from the shape. The design enabled access to obscured features of 3D shapes e.g. the ‘hidden corner’, when viewed from particular orientations. Touch provided access to ‘innards of objects’ (Minogue and Jones 2006): here, internal exploration of the cube was enabled through tactile sensation elicited via force feedback from the inner surface and corners of the cube.

Tasks were designed based on the UK national curriculum for year 2 (6–7 years of age):

identify and describe the properties of 2D shapes, including the number of sides; identify and describe the properties of 3D shapes, including the number of edges, vertices and faces.

For year 3 (7–8 years of age):

draw 2D shapes and make 3D shapes using modelling materials; recognise 3D shapes in different orientations and describe them.

In SpaceShapes, there are three different tasks.

Task 1

The initial screen shows a moonscape with an invisible cube, that children have to reveal by using the haptic device to explore and identify the shape. The first corner is visible to give a clue as to where to move the haptic device in order to find another corner. Touching the shape at its edges and vertices produces the wireframe of the cube. In order for an edge to appear, the students need to touch the intersection of two faces, which belong to the same edge (one after the other), essentially running the stylus along the ‘felt’ edge of the cube (kinesthetic).

When two consecutive corners of the cube are touched, the edge connecting them appears on the screen, and when all four corners of the same face of the cube are identified, then this face appears coloured (see Fig. 3).This design not only fostered tactile experience of features of the cube, but also kinesthetic experience through specific hand and arm movements tracing the edges of the shape (https://tinyurl.com/ruj77y3). Teachers can configure how many corners are visible at the start, depending on the desired level of challenge for the children.

Fig. 3
figure 3

Process of making the invisible cube visible using the haptic device.

Full size image

Through the lens of epistemological domain (Balacheff and Sutherland 1994), the learning design aimed toexploit the potential to structure systematic haptic exploration around the mathematical properties of the cube (i.e. the relationship between vertices), limiting random movements of the haptic device. This was achievedby supporting students to make conjectures about how to move the haptic device from one corner to the next, taking into account the position of the visible vertices and shape of the emerging cube.

The progressive generation of the cube aimed to support students to take into account properties like parallelism, depth and relationships among the different vertices, edges and faces. Children were supported by: (a) haptic feedback which enabled them to feel the cube, through a sensation of resistance, like a wall, and loss of resistance as they go over an edge or a corner; (b) information relating to cube properties (moving alongthe face or edge of the cube); (c) visual feedback as a result of student haptic actions (i.e. a corner appears when it is touched), providing information to support further haptic exploration. This task makes the properties of the cube instruments for its construction, and grounds the construction process on complementary sensory information through an interplay between visual and haptic feedback.

Task 2

The second task is a colour-coded mapping task to facilitate correspondence between faces on the 3D shape andits 2D representation. When children connect the relevant faces, the face of the cube opens,and the haptic device can be used to manipulate the box, to ‘tip out’ one piece of rocket and to collect rocket fuel cells from the four internal vertices (see Fig. 4). The learning design aimed to facilitate perspective taking through manipulation of the cube, to foster correspondence between a 3D shape and its 2D representation with an emphasis on faces and squares and their position in the 2D and 3D space, and – of specific relevance to this article –to bring visuo-tactile awareness to the hidden corner, through a sensation of resistance encountered when collecting the rocket fuel cellsfrom the inner corners of the cube, similar to the sensation of a pen being moved around the inner corners of a physical box.

Fig.  4
figure 4

Manipulating the box to tip out a rocket piece and collect fuel cells from internal corners

Full size image

Task 3

The third task involves assembling the rocket by putting its pieces together, one on top of the other, which does not exploit the haptic properties of interaction. Thus,the focus of this article is on children’s interaction in Tasks 1 and 2.

Methodology

This qualitative exploratory study took place in the children’s everyday classroom. It drew on video data of paired children’s interaction with a purpose-built digital learning experience to support children’s engagement with 2D and 3D shape through a visuo-tactile interaction using an OmniPhantom haptic device, and follow-up interviews. Within the embodied cognition paradigm, “the unit of analysis for understanding cognition is the brain-body-environment in their dynamic interplay” (Gallagher and Lindgren 2015, p.394).

Thus, the study aimed to understand how children’s in situ exploration and communication of geometrical concepts of 2D/3D shape was mediated by the haptic environment to answer the research questions: how can haptic interaction foster specific action experiences that shape the way that children conceptualise 3D shape? How do these experiences shape children’s communication of ideas around 3D shape? How do children use their body as an instrument for participatory sense making while interacting with the haptic device?

The analysis focused on the in situ dynamic unfolding of sensorimotor interaction of pairs of children as they engaged with mathematical ideas around 3D shape with a visuo-haptic environment, and their communication of ideas following interaction.While the research design fosters paired collaborative interaction, the focus of this analysis is on the role of sensorimotor interaction with the haptic tool, in shaping children’s engagement with and conceptualisation of shape, part of which draws on their verbal and gestural communication. In so doing, it aims to gain a better, in-depth understanding of how the digital environment shaped children’s interaction and the way their sensorimotor experiences underpinnedcommunication in relation to mathematical ideas.

Participants and Procedure

Ten pairs of children (n = 20) aged 7–8 years from a UK primary school participated in an in situ, constructive dyad interaction design to encourage collaboration and communication (Als et al. 2005). They were selected on the basis of having provided parental informed consent, giving their assent prior to participation and being made aware of their right to stop (withdraw) at any time. They were paired on the basis of who the teacher considered would interact wellwith one another. No children in this class had engaged with a haptic device before.

Each pair were informally interviewed beforehand to establish their experience with computers and computer games (home and school use), whether they had experienced games using bodily forms of interaction (such as the wii), and what they could tell us about 2D and 3D shapes. One pair at a time interacted with the environment,seated at a table with the laptop computer and OmniPhantom. The children were informed that the researchers were interested in how they used the haptic device to explore mathematical ideas, and whether or how it helped them think about the ideas, and that the device allowed them to feel the texture and shape of things. They then took turns to use the device in ShapeBalloons to familiarise themselves with the movement of the device in 3D space, and then in SpaceShapes.

A researcher provided support when children were experiencing difficulty, to keep them on task (e.g. can you find any more corners of the shape?) and draw attention to haptic sensation where needed (e.g. can you feel the side of the cube?).Primarily, support was provided through naturalistic verbal interaction, sometimes accompanied by deictic gestures toward the screen. On completion of the tasks, each pair took part in a semi-structured interview focusing on their experience with the haptic device, what it felt like (tactile experience), how they used it to reveal the shape, how they conceptualised the properties of the cube, and how they would describe what to do to a friend who was going to undertake these tasks. Each session, including the interview, lasted around 45 min.

Data Collection and Analysis

Data collection comprised video-recorded data of pre- and post-interviews and interaction with the environment. Two video cameras recorded the interaction: one to the side and slightly behind the pair of children to capture the screen and their hand and arm movements, both with the OmniPhantom and in gesture, the other situated facing the pair of children to capture communication and action.

Analysis of the eleven hours of video data of pre-interview, interaction and post-interview focused on: (a) haptic interaction examining children’s tactile or haptic experience relating to 3D interaction in 2D representation; (b) the use of the body as an instrument for participatory sense-making, looking at enaction, language, gestures and embodied experiences; (c) identifying strategies children used for engaging with the mathematical ideas (3D space in a 2D screen and properties of 3D shape). A multimodal analytical approach (Jewitt 2009) directs observation of video data on action, gesture, gaze and speech, bringing the body into the focus of the analytical frame. This approach is beneficial in identifying how the body is brought into dialogue with mathematical ideas (e.g. relevant action repertoires, tactile and visuo-tactile experiences in relationship to interaction in 2D and 3D space), noting the relationship between embodied action and language, and identifying the use of spontaneous representational gesture (Hostetter et al. 2007; Thelen et al. 2001).

Three researchers analysed the data through a repeated and iterative process of viewing video recordings. Firstly, patterns of interaction and communication across participants were identified and, through discussion, formed the basis of the key themes emerging. Secondly, key thematic episodes were identified and transcribed from each data set,noting gesture, action and speech, including screen shots of relevant action or gesture. Finally, data for each theme was collated to provide evidence for the findings.

Findings

The aim of this study was to understand better the process of student learning with a haptic device. Specifically, we examined: (a) how it mediated interaction, the kinds of action experiences and action schemes it elicited (how can haptic interaction foster specific action experiences to support learning around 3D shape?); (b) how it mediated communication (how do these experiences shape children’s communication of ideas around 3D shape?); (c) the strategies children collaboratively developed to complete activities, and evidence for relating their body-based interaction to mathematical concepts e.g. 3D space and shape, spatial awareness, features of shape (how do children use their body as an instrument for participatory sense making while interacting with the haptic device?).

Working in pairs, one child at a time used the haptic device, while the other offered guidance and support in the tasks assigned. Four overarching themes related to the research questions emerged from the analysis and are discussed in the light of embodied cognition theoretical perspectives:

  • conceptual metaphor: body-based language development;

  • haptic sensory interaction;

  • pointing, gesturing and external props;

  • enactive metaphors for gestural communication.

Conceptual Metaphor: Body-Based Language Development

The analysis showed how children’s language developed, becoming enriched and more specific in relation to 3D interaction through the combination of the haptic environment and the collaborative nature of the students’ activity. Some found the initial interaction with the haptic device challenging, in particular manipulating or moving it in relation to the visual representation of the purple ball (cursor) on the 2D screen. While they soon mastered the manipulation, these initial periods of interaction elicited insights into children’s perception and bodily experience of 3D interaction on the computer through their verbal and gestural communication.

Their use of body-based metaphors (e.g. up, down, forwards, backwards) underpinning their verbal communication were instrumental in mediating action in 3D space, particularly along the z-axis. Evidence for this emerged initially through episodes where verbal expression conflicted with expected action, which aligned with more familiar mouse movement linked to 2D screen representations. These conflicts led to extended and more precise linguistic use that supported effective movement in the 3D space.

Conflict between Language and Expected Action

Conflicts in communication of movement in 2D vs 3D space revealed insights into the children’s developing sensorimotor perception of and interaction in 3D space, specifically through action verbs related to movement along the different axes. For example, in Space Balloons, the participant Oscar guided his peer to locate a particular balloon, and said,“which is… just, just go a little bit up [pointing] … a tiny bit”,where ‘up’ was used to reference movement along the z-axis, i.e. referring to moving the cursor ‘along’ the track away from themselves. This required a forward or ‘push’ movement of the device. However, the use of ‘up’ in this context was potentially confusing, since it could also refer to moving the cursor ‘up’ the string to pop the balloon, mapping to an upwards movement on the 2D screen, akin to mouse action.

The same pair encountered further challenges with verbal instructions that map to forward and backward movements on the z-axis. This arose since a push forward action on the haptic device leads to a backwards movement of the cursor on the screen, i.e. it goes ‘deeper’ into the 3D space. In contrast, a forward movement of the cursor on the screen requires a pull-back action on the haptic device. For example, when guiding, Oscar said: “go up…go up...now go forward…move back…and down…down…then go to the right… right…up up up up…back back”, prompting movement with the haptic device that was opposite to that intended in the instruction. This mis-match between the screen and the haptic device led to some confusion for Oscar.

In this instance, the researcher intervened with, “I think his back and forward is different…so maybe try saying forward”. This example illustrates challenges in describing 3D movement on the z-axis – which is forwards with hand movement to activate backwards in the digital environment, and vice versa – for children familiar with mouse interaction in 2D space.Children’s verbal and gestural language illustrated how their bodies’ sensorimotor functioning provided a foundation for developing communication of specific movements that support 3D interaction in conjunction with the 2D screen.

Language Supporting Effective Movementin 3D Space

As the interaction progressed, instances where the guiding peer used action verbs (often accompanied by gestures) that supported movement along the z-axis increased, both in SpaceBalloons and in SpaceShapes. The most commonly used verb was ‘push’, to indicate a movement deeper along the z-axis, and the emerging successful use of ‘forwards’ in reference to the device, again moving the cursor deeper along the z-axis,and ‘backwards’ to bring the cursor nearer in the 3D space (see Table 1).

Table 1 Use of action verbs to support movement along the z-axis
Full size table

As interaction progressed, peer descriptions of movement became richer, more precise in terms of the different directions to move along each of the three axes. Since haptic device interaction with screen technologies involves moving along the x-, y- and z-axes, it contrasts with mouse interaction which typically comprises movement of up, down and across, but not forwards or backwards. Using the OmniPhantom requires a different conception of how to move in order to place the cursor in the desired location.

Children’s participatory sense making (through the haptic interaction) elicited language and gesture that referred to 3D space, demonstrating their perception of depth – z-axis – expression of 3D interaction in the 2D space. This not only illustrates the changing role of body-based metaphors in conceptualising space in conjunction with the screen representation (from mouse interaction), but also demonstrated children’s spatial awareness and ability to identify position in 3D space. The use of the Omniphantom (in contrast to a mouse) thus enabled children to gain sensorimotor experience of thez-axis, as well as the x- and y-axes. In addition, their developing verbal expressions to support relevant movement trajectories showed how they conceptualised this movement.

Haptic Sensory Interaction

The use of tactile feedback was central to SpaceShapes, where children had to use the haptic device to locate the vertices of the cube (tactile) and move along the edges of the cube (kinesthetic), in order to make the cube visible. Across the interaction process, children’s attention focused on the haptic feedback itself, as well as on the cube’s structure and properties. In this sub-section, we give examples that demonstrate children’s engagement with the haptic feedback.

In the early stages, children moved the haptic device randomly, in several cases serendipitously making one or two vertices appear. However, after some vertices were found (e.g. 3 or 4), children started conjecturing about the position of the next one. In this case, movements of the haptic device became intentional, slower and more careful,engaging with the‘felt’ edge of the cube and manipulating the cube to bring it into a different persective (see Fig. 5).

Fig. 5
figure 5

Manipulating the cube to bring the target corner to the front of the screen: (a) touching the top face of the cube: (b) turning the cube 90 degrees along the x-axis: (c) turning the cube 90 degrees along the y-axis

Full size image

When trying to reveal the cube, children initially appeared to struggle to make the connection between the tactile pushback from the haptic device and the presence of the cube. The researcher facilitated children’s interactions and understanding of what these sensations related to, enabling children to be more mindful of how the tactile element could facilitate their completing the task. For example, Nathan said, “It’s so stiff”. In response to this the researcher said, “It will feel stiff because the cube, although you can’t see it, when the ball bumps into it, it won’t let you go through, so when you feel it go all funny, that’s because you are bashing into the side of the cube”.

When the haptic device was not touching the cube, it moved freely in 3D space. The edge was captured when the device met the resistance when pushing hard, and their gesture movements became slow and steady. When they moved beyond the edge of the invisible cube, the child’s hand suddenly moved forwards since there was no longer any resistance. This resulted in children trying to stay on the edge of the cube,using other resources to help them, e.g. moving the cursor parallel to other visible edges (see Fig. 6) (see the next sub-section).

Fig.  6.
figure 6

Step by step illustration Nathan tracing edge of cube with haptic device: (a) moving the HD slowly parallel to one visible edge (top face); (b) the movement continues until the cursor reaches the point where two edges meet – at this point, the lines connecting two consequetive edges appear and the face becomes coloured; (c) when all vertices of the same face are found the whole face becomes opaque

Full size image

A similar interaction was also observed with Anna and Sophie:

Researcher: Can you feel the side of the cube there?

Anna: Er, a bit, yeah

Researcher: So let’s try that again

Anna: I can feel it, oh I think we’re crashing into it

Following these interactions, the children became more aware of the haptic feedback and its function. In the post-interaction interviews, Anna said, “you’d have felt it, but if you were using your fingers [referring to non-haptic technology situation] you wouldn’t have”. Another child (Hilary) described the sensation of receiving pushback from the haptic device, showing her expression of the haptic feedback through using her body. With her hands clasped together and shaking her body, she said, “when I gone in a bit far, it was like …”.

Collectively, these examples suggest that while the haptic feedback is an affordance in the design – in that children are able to infer the location of the cube’s corners through tactile sensation –it also fostered engagement with shape in a more sequential way than visual representations alone.This resulted both from the design of the task and from the interaction with the haptic device, which demands one point of contact, fostering sequential movements (Minogue and Jones 2006). In so doing, the device focuses children’s attention on the structure and properties of the cube and, in particular, on decomposing the features of the shape in a non-iconic way (Duval 2005).

Extensive research shows the educational benefits of simultaneous presentation of information available in visual representations (as opposed to the sequential processing involved in written and spoken text) (Larkin and Simon 1987; Price and Scaife 2002). In contrast, this study suggests the important role of the tactile (multisensory) engagement in enabling support for dimensional deconstruction (Duval 2005) through sequential interaction with a visual representation, also shown to shape the later enactive communication of 3D shape (see the final sub-section). Although the children needed some support from the researcher in order to make the connection between the haptic feedback and its referential meaning, this could be due to the novelty both of the task and of the haptic device. However, with more exposure, the potential for more independent tactile interaction could be increased.

Pointing, Gesturing and External Props

The haptic environment supported the use of various strategies – beyond the tactile – to locate the vertices of the cube. This was evidenced through episodes of non-verbal communication, where children used other bodily resources for communicating about the geometrical ideas around the visual and tactile interaction, including pointing, gesturing and using external props, which contributed to their participatory sense making (van Dijk 2018). Typically, the child who was not holding the haptic device engaged in this form of communication when making suggestions to their peer.

As an illustrative example, Oscar indicated the proposed trajectory – the direction of movement the haptic device should take – by moving his finger along a pathway on the screen (Fig. 7a). This pathway was identical to that of the cube edge. Oscar noticed, and made use of, the shadows of the corners of the cube (Fig. 7b) reflected on the ground, to conjecture the position of another corner of the cube, to point out the proposed position of the corner on the screen. Both of these strategies were instrumental in children collectively identifying edges and corners that would reveal the cube.

Fig. 7
figure 7

(a) Oscar pointing to the top corner where the HD needs to start from, before moving his finger down towards the ground parallel to the edge on the left (perpendicular to the top face). (b) Oscar pointing to the shadow of the cube to map the proposed position of next point

Full size image

These conjectures that occurred across pairs suggest that the children were using mathematical ideas to inform the use of the haptic device, as well as developing an ability to decompose the features of the cube. Specifically, it demonstrates their developing understanding of the topology of the cube, through conjectures related to spatial relations – the proximity, distance and direction, as well as properties of the cube, its number of vertices and edges.

One pair made use of other external resources – here,a ruler –to inform and guide themovement of the haptic device. It was placed in parallel with another edge of the cube, illustrating that the child (John) recognised that some edges are parallel, and supporting intentional movement of the haptic device. He also pointed to where he thought one of the corners would be located (Fig. 8 includes a step-by-step breakdown).

Fig. 8
figure 8

Step-by-step illustration of using an external resource and embodied dialogue to support interaction with the game: (a) John aligning the ruler to the screen; (b) John placing the ruler parallel to the top edge of the cube (line annotation in the image); (c) John, having removed the ruler, placing his finger at the position indicated by the ruler and aligned with the top corner

Full size image

These examples illustrate ways in which children used the visual cues to guide the directional movement of the haptic device, while simultaneously enabling tactile experience of the 3D cube edges. This analysis provides evidence of children’sspatial ability in relation to 3D shape on the 2D screen, through their predictions of the location of cube corners, since these were often offset from the 2D square shapes which form the faces of the cube. Combined with the visual, the haptic explorations differently supported awareness and identification of the distinguishing features of objects, extending similar findings with older children using haptic interaction (Alexander et al. 2002).

The distinguishing features of the objects also shaped the use of the haptic device, such that it became an instrument for the children with which to complete the task. Analysis shows that the haptic interaction both informed the location of vertices, corners and the distance between them (through tactile interaction) and was informed by other resources that drew on visual information, generating a kind of ‘mathematically informed touch’, or on the use of mathematical ideas to shape their touch action trajectories.

Enactive Metaphors for Gestural Communication

Aligning with an enactive approach (Gallagher and Lindgren 2015), the design of the haptic experience fostered specific action repertoires that enabled children to trace a 3D shape through sensorimotor engagement: specifically, using their arm and hand. These show how the touch interaction and associated guided movements through the haptic device brought attention to mathematical properties of the cube, beyond notions of materiality. Development of these action repertoires were particularly evident where the task was to reveal the invisible cube through tactile engagement with the edges and corners of the shape in conjunction with visual cues to which they were related.

Action repertoires were evident in two ways: direct action through interaction with the haptic device and gesture-based action from the collaborating peer. For example, the collaborating child’s gesture involved enacting the direction the haptic device should move along the edge of the cube, rather than merely pointing to a predicted position of the next corner. This suggests that at times the children’s tactile exploration of the cube was informed by their conjectures about properties of the cube, both the edges and the corners. These action experiences seemed instrumental in shaping the way children described the cube in the post-interaction interview, as the children used theseaction trajectories as key elements in their re-constructed representation, illustrated through spontaneous representational gestures in their communication. Here we provide three illustrative examples.

The first shows how one child (Sean) enacted the generation of the cube based on its corners (Fig. 9). Hisindication of the cube consisted of pointing gestures of the positions of six out of the eight corners of the cube and, in so doing, also showed the edges between each of the corners that result in a cube’s shape. Looking back at the interactive experience, this is interesting because it follows the way the cube was generated in the game (based on the corners rather than on drawing squares or the edges), showing how the specific experience underpinned the child’s recreated model – in this instance,emphasising the corners of the shape. This also shows that the game design foregrounded the corners in the child’s attention, which not only demonstrates their conception of the 3D shape, but also addresses issues of remembering and recognising the hidden corner of the cube.

Fig.  9
figure 9

Focus on corners and the trajectory of Sean’s gestures to form a representation of a cube

Full size image

Similarly, when asked how many corners there were on the cube, Anna spontaneously gestured to support her calculation of the number of corners in the cube. “Six…wait no… [outlining the cube] …1, 2, 3, 4, 5, 6… oh wait …”(Fig. 10, child with hair tied up). Following further discussion with the researcher and additional gestures, the child then said, “one, two, three, eight”. Again, this illustrates the role of the action trajectories in interaction, in underpinning her ‘thinking through’ the properties of the cube, and supporting an effective way to communicate her ideas about 3D shape and relative spatial positioning of corners.

Fig.  10.
figure 10

Step-by-step ilustration of Anna counting the corners of a 3D cube shape in 3D space

Full size image

In the third example, when describing the learning experience to the researcher, Iris described the shape of the cube using representational gesture that depicted the outline of the cube in 3D space (Fig. 11). She said it felt “ a bit different than holding a 3D cube in your hand because like you have to imagine how you would feel it in your hand and like try and draw it with a pencil.” Here, the pair re-enacted the trajectory of motion used in revealing the 3D cube, suggesting the salience of haptic device in 3D movement in providing the underpinning of this sensorimotor representation used during explanation, and the role of Task 1, in its shaping of these actions through construction of the cube based on key features, such as, predicting and identifying edges.

Fig.  11
figure 11

Iris re-enactingthe cube during the post-interview.

Full size image

These examples demonstrate how specific action experiences, mediated by the haptic task, underpinned and shaped the way children then communicated their understanding of a 3D cube. The action experiences in this study were shaped by embodied participatory sense making, i.e.through the joint completion of the task that specifically involved using the haptic device to find vertices and edges of the 3D shape that fostered particular actions that map to the enaction of drawing a 3D shape.

Further, our findings demonstrate that students gradually developed a form of mathematically informed touch, a form which evolved along with the construction of the cube. Initially, it involved a random movement of the device. As soon as the first few vertices appeared, the use of the haptic device became more deliberate and involved participatory sense-making which included action-oriented, spatial-verbal communication (up, down, forwards), use of mathematical external resources (a ruler) and mathematically informed gestures (indication of trajectory parallel to existing edges, consideration of the shadow of vertices and pointing). The force feedback of the cube provided information in relation to what is ‘cube’ and what is ‘not-cube’, emphasising the kinaesthetic aspect of touch (i.e. the children finding the vertices by moving the cursor along the edges). This crafted movement resonates with specifically adopted dragging modalities by students in dynamic geometry (Arzarello et al. 2002).

Discussion

Embodied cognition foregrounds the role of bodily experience in shaping cognition. When children are learning about shape, and 3D shapes in particular, they typically engage with physical models of 3D shapes (e.g. cubes, polygons) or work with 2D representations of 3D shapes. While these are valuable resources, some challenges arise, in particular in relation to the ‘hidden corner’ on 2D representations of a cube, as well as properties of the cube in general. Haptic technologies enable new ways of experiencing 3D shape within a digital context, typically linked to a screen-based visual representation.

In this study, we aimed to analyse how haptic interaction fostered specific action experiences and shaped the way that children conceptualise and communicate ideas around 3D shape, and how children use their bodies as instruments for participatory sense-making while interacting with the haptic device. We observed how they made spatial conjectures about shape features based both on tactile and on visual elements, and illustrated how the haptic interaction brought children new sensorimotor experiences when exploring 3D shapes and 3D space. Changing the modality of interaction thus enabled “new forms of embodied mathematical activity” (Hall et al. 2014, p. 113) and opportunities for “new capacities to feel and touch aspects of mathematics” (p. 114).

This brought new sensorimotor experiences for children to draw on, for example, sensing and moving along edges, vertices and corners that form the key properties of shapes, creating sensorimotor pathways related to geometrical learning, and experiencing unfamiliar, partial views of the shape, as well as experience both of inside and of outside the shape (Davis et al. 2017). In this section, we discuss these findings in relation to participatory sense-making and how it relates to conceptual and enactive metaphor, and the role these play in children’s developing mathematical understanding and awareness, demonstrated through communicative gesture.

Drawing on notions of participatory sense-making (De Jaegher and Di Paolo 2007), the findings show ways in which this process supports children’s developing understanding from an embodied cognition perspective. We illustrated three key processes underlying this development and the role they played in shaping children’s conceptions illustrated through their gestural communication. Firstly, the haptic device and their collaborative activity provided image-schemas based on sensorimotor experience, specifically, directional movement of the hand and arm manipulating the haptic device, that resulted in changes in how children described ‘movement’ in the 3D space.

The language the children used showed how their conceptualisation of movement in 3D spacechanged from the familiar ‘up, down, across’ movement in a 2D screen context to ‘forward, backwards, up, down, push, pull’ movement in 3D space in conjunction with the 2D screen. This resonates with notions of conceptual metaphor, where the bodily engagement with the haptic device provided a foundation for conceptualising 3D movement on the 2D screen.

Secondly, the multisensory engagement – the combined tactile and visual – was shown to mediate new forms of interaction and thinking. In particular, the study shows how the haptic interaction provided (sensori)motor interaction along the z-axis, as well as the x- and y-axes. Thus, not only did the children ‘see’ the 3D space through the visual presentation, they also moved their bodies (in this instance, their arms and hands) in 3D directions, providing multi-sensory (tactile, kinaesthetic and visual) experience of depth within the 2D screen context.

The tactile provided something ‘physical’ in terms of sensation – acting as a form of physical anchor, that then linked to the visual representation of the 3D object, and across the task, resultingin the sensorimotor action that underpinned the shape, or a simulation of drawing a 3D shape. Through this, the haptic experience provided new ways to foster non-iconic engagement with 3D shape (Duval 2005), important for reasoning about geometric properties, and to conceptualise 3D shape in ways that fostered the development of 3D body- based experience in relation to 2D screen interaction.

According to Gallagher and Lindgren (2015),“new technologies are capable of delivering perceptual cues that elicit expert actions” (p.399). However, the study reported here shows how they can also provide new ways of interacting, which fosters new action trajectories and ways of tracing 3D objects, giving action access to 3D perspectives and hidden parts of objects (Davis et al. 2017), such as the hidden corner. While information about 3D shape was shared across visual and tactile interaction, a key aspect of the haptic was in supporting precise action trajectories that aligned with 3D object edges, and in relation to perspective in 3D space.

Our findings show how this experience provided important sensorimotor contingencies that children used later in talking about their experience, evidenced through their gestural communication about creating a cube shape, thus extending previous work in enactive cognition (e.g. Gallagher and Lindgren 2015). These experiences were shown to underpin enactive metaphors for gestural communication, andto reflect the link between the prescribed activity (through the design of the environment) and sensorimotor patterns that underpin the conceptualisation of 3D shape, and specific features of the shape, that the children used to communicate through gesture (Reiner 2009).

In this way, the bodily experience or ‘lived’ moving body (Merleau-Ponty 1962) of the haptic device has been incorporated into a way of thinking about geometric concepts (in this instance, 3D shape) (van Dijk 2018). We can think of the haptic as an active component within the children’s mathematical activity, where the tactile experience complemented and reinforced visual perspectives, and provided the means for meaningful movement in relation to 3D space and 3D shape. “Not only do artefacts acquire meaning courtesy of being situated in social practices; the same artefacts play an active role in the formation and sustaining of said practices” (van Dijk 2018, p. 9). Indeed,“the development of a sensorimotor coupling can be seen as the development of a ‘skill’”(p.11). From the findings, in particular the gestural communication, we could argue that the haptic interaction provided the skill that underpins the ability to ‘draw’ 3D shape.

Conclusion

This study examined the role of a haptic learning environment in supporting 7–8-year-old children’s embodied exploration of 3D shape. Specifically, it analysed the process of how the prescribed enaction of the device mediated interaction, the kinds of action experiences and action schemes it elicited, and the strategies the children collaboratively developed to complete tasks. In so doing, it provided evidence for the relationship between sensorimotor interaction and mathematical concepts of 3D space and shape, and the role this interaction played in underpinning children’s use of representational gesture when communicating about their learning experience.

The findings show how specific action experiences, and later communication experiences, were shaped not only by the prescribed enaction of the design, but also by embodied participatory sense-making. This was evidenced in three key ways. Firstly, through changes in children’s language and gesture that reflected the changing role of their body-based metaphors in conceptualising 3D space in conjunction with the 2D screen representation, demonstrating developing perception of depth on the 2D representation. Secondly, the haptic feedback in the taskdesign provides one point of tactile contact, fostering sequential movements with the device which was shown to foster dimensional deconstruction of shape, and to underpin later enactive communication of 3D shape. Thirdly, the haptic interaction combined with the visual provided new and different ways for children to identify and engage with key distinguishing features of objects.

However, two limitations are of note. Firstly, as with much research on novel technology environments, empirical work involved limited use of the haptic device, i.e. one session lasting some 45 min. While this has significant implications for interpreting results in terms of engagement and usability, it did provide the foundation for in-depth analysis of sensorimotor interaction and implictions for meaning-making. Nevertheless, it would be interesting to examine multiple haptic interactions over time, to develop sensorimotor-cognition understanding further, and how or if patterns of interaction change over time, and their relationship to children’s mathematical concept development. Secondly, the analysis revealed the foundation that the haptic device provided for structuring gestural communication of 3D shape. Future work could include other methods for eliciting such information, such as drawing 3D shapes, to examine the relationship between this process and the haptic interaction, as well as the outcome (i.e. the drawn shape).

This is one of the first studies of haptic interactions with elementrary-school-aged children that examines the learning process from an embodied cognition perspective. In so doing, it makes a theoretical contribution providing evidence for the ways in which sensorimotor interaction underpins conceptualisation, a contribution to technology design, particularly how design shapes human action and interaction, and a contribution to digital learning, demonstrating the potential for haptic technologyin mediating new learning experiences for mathematics.