1 Introduction

Sonification of movement has been shown to be beneficial for motor-learning, sport and rehabilitation [1]. However, as we continue to develop technology to support motor-learning/rehabilitation, it is important to fully understand the impact sound-based technology has on the movement itself, including the opportunities it offers. Recent work has begun to explore how changes in movement behaviour arising from movement feedback can be used to alter that movement behaviour implicitly [2, 3]. For instance, it has been found that the inclusion of musical structure within sonification can change the way people move [4, 5]. This is likely related to the implicit and embodied way in which people understand music [6, 7], meaning that even non-musicians can perceive an embodied connection between their own body movement and a musical stimulus. The use of musical stimuli during exertion exercise has been shown to both lower people’s perceived exertion and allow them to continue longer before exhaustion [8]. Moreover, movement-altering sonifications have explored a number of different sound types and their effect on both movement performance and bodily perception [9]. However, how these sonifications develop an expectation of movement which can then be manipulated through sound is yet to be investigated.

This may be particularly useful for people with low self-efficacy, i.e. limited belief in one’s own ability, who may struggle to engage in physical activity [10]. Bandura describes how the feeling of self-efficacy is one of the core factors involved in whether someone will complete an activity [11]. People’s perception of their own behaviour, overcoming obstacles and achieving goals are all guided by their own belief in their ability to do so. Fitzsimmons et al. show how self-efficacy can be used to impact the performance of weightlifters [12]. Through providing altered performance feedback (people were told they had lifted more or less than they had) the impact of this feedback on people’s performance and perceived self-efficacy was measured. It was found that participants could lift more when they were told they were lifting less, though this was regulated by previous experience. This demonstrates how by altering the feedback people’s performance can be altered, as well as showing the strong impact that one’s belief in one’s ability has on what can be achieved.

Schunk describes both performance feedback (which informs the performance of a task) and attributional feedback (which aids in attributing that effort to successes). For example, being told, “you have been working hard”, as opposed to “you will have to work hard” was found to lead to higher motivation and self-efficacy in children learning mathematics [13]. By supporting the attribution of success and offering encouragement to continue on, the barriers of low self-efficacy can be overcome. This perspective considers not only the need to inform people of their ability so as to understand their capabilities, but also the need for encouragement in the face of obstacles to help increase those capabilities. In addition, it has been shown that providing encouragement to continue past one’s perceived limits can help support populations with low self-efficacy [14, 15].

Musical structure has been used to improve the aesthetics of a sonification [16], to improve the efficacy of its ability to inform [14, 17, 18] or to promote engagement [19]. Many authors have begun to consider the differences between sonification and music [16, 20, 21]. In these considerations, they warn of the dangers of conflating “informational sonification” and “aesthetically pleasing music” and call for sonic information designers to take care when including music within sonification design, not to distort or obfuscate the underlying representation of the data.

However, with the rich implicit and embodied understanding people have of music, there is a potential missed opportunity in overly limiting its use with sonification. Understanding the way in which the use of musical structure impacts one’s perception of a sonification (and in the case of movement sonification, one’s behaviour) is essential for designing understandable and effective sonifications. Our musical and embodied understanding of sound may change the way we perceive and interact with these sonifications [16, 21].

In this paper, we conjecture that movement-altering sonifications affect movement by altering one’s mental model of “movement expectancy”, which can be used to support low self-efficacy. This motivates the development of a Movement Expectation Sonification Model (MoSEM) based on Huron’s theory for psychological expectancy in music [22] but considering the impact of using these musical stimuli as feedback on one’s own movement [23]. We then evaluate this model and explore how it can be used to describe the impact of Music Expectancy Sonification (MuES) on people’s movement and perception.

The paper makes three contributions:

  • A model for expectancy-based movement-sonification design based on Huron’s musical expectation theory.

  • An evaluation of the movement expectancy model and how it may be developed.

  • A demonstration of how such movement-altering sonifications are mediated by cues in the environment.

2 Background

Sonification has been used to provide movement information in a variety of applications. In sports and rehabilitation, movement sonification has been used to support the optimisation of movement and the learning of new movements (see Schaffert et al. for a review [1]).

In addition to using sound to inform people on their movement, recent work has explored how sound feedback may be used to alter one’s bodily representation and movement behaviour. The sound manipulation chosen for movement feedback has been shown to be able to alter both people’s perceptions and movement behaviour (see Bevilacqua et al. for a review [24]). Tajadura-Jiménez et al. show how altering the frequencies in a person’s footstep sound changes the way they walk and their body perception [2]. People had more energy and felt lighter when they heard higher frequencies. Boyer et al. show how longer sounds can lead to longer trajectories for hand movements, as well as improving precision [3]. This work shows that, even outside of music, sound can have a powerful impact on altering movement behaviour.

In addition to altering the sounds of bodily feedback, people’s embodied understanding of music [7] allows for the musical structure to be used to alter movement. Music has been shown to encourage movement and provide motivation during exercise, and the use of musical stimuli has also been shown to allow users to continue longer before exhaustion [8, 25]. Dyer et al. compared melodic versus rhythmic based sonifications and found that not only did the melodic sonification have greater immediate retention, but also that after 24 h, a replay of the melodic content allowed participants to improve their performance without the use of the concurrent sonification [5]. This work demonstrates that the use of musical structure cannot only aid movement retention but demonstrates how our embodied relationship with music can be utilised within sonification. Newbold et al., builds on the work of Singh et al. and Wallis et al. [14, 17] by showing how the sonification of target points in the stretch-forward movement can use harmonic resolution to alter movement behaviour subconsciously [4]. By mapping the completion of a stretch-forward movement to either a musically complete ending or musically incomplete ending, they found that the tension/resolution of the music could not only be used to motivate/reward the movement but that it also affected how people mapped the completion of the music to the completion of their movement. These sonifications offer a way to provide both performance feedback (through sonification) and attributional feedback (through music) [13]. However, while these sonifications have been shown to be effective, there is little examination of what theoretical model may be guiding the effect of musical expectancy in sonification, nor how the sonic cues interact with additional cues in the environment that may signal the end of the movement (and potentially render the sonification less effective in combination).

3 Musical expectancy in sonification

In order to understand how musical expectancy is experienced in sonification, we must first explore the embodied experience of music. People understand many aspects of music through mere exposure to it in day-to-day life and have the ability to identify musical grammar without formal musical education [6, 26]. Listening to music on the radio (and in other areas of day to day life), people build an understanding of how music works, and through the implicit learning of these musical “rules”, form an expectancy of how a piece of music should continue. It is the manipulation of this expectation that creates the tensions and relaxations felt in listening to music. Additionally, this relationship with music has been linked to the embodied way people understand many of the underlying structures of music [7, 27], for instance, one may talk about how a piece of music “moves through a progression”.

Huron builds on this understanding of musical expectation, defining five expectation-related emotional response systems: ITPRA theory [22]. These five responses are as follows: Imagination response (where the expected outcome is established), Tension response (where the arousal is heightened as the person awaits their expected conclusion), Prediction response (where the prediction of expectation is evaluated as either correct or incorrect), Reaction response (where the result of that evaluation is unconsciously reacted to in the short term) and Appraisal response (where the result of the evaluation is consciously judged in the long term). This model gives a clear structure to the way in which people react to different manipulations of musical expectation.

Musical expectancy has been demonstrated to impact people’s movement at given target points. The work by Komeilipoor et al. [28] showed how musical dissonance impacts people’s ability to synchronise their movement with an external musical stimulus. Participants moved their finger between two points while listening to musical stimuli. It was found that participants were able to better synchronise with a consonant sound (one that fits expectation) than a dissonant sound (one which defies expectation). Additionally, it was found that the consonant sound improved both form and accuracy. This shows how people’s movement was changed to fit the musical expectation when the sound was consonant and complete. It encourages the listener to complete the movement in turn. When dissonant sound implied to the listener that the music had not reached the expected conclusion, they were less able to synchronise the movement. The work of Wallis et al. and Huang et al. both show how harmonic dissonance and resolution can be used to reward correct movement and punish incorrect movement [17, 29].

Newbold et al. show how people are encouraged to move further when their expectation is defied and feel a sense of completion when their expectation is met [4, 30]. These works all demonstrate how these musical constructs can be used within sonification to alter one’s perception of one’s own movement. However, it has yet to be explored how such expectations can be manipulated within movement, or the external factors that may play a role in how such an expectation is built. For example, how can current theories of musical expectation and movement feedback be used to explore how one’s “movement expectation” can be manipulated, by considering the individual’s active role in the generation of musical sonification.

3.1 Movement Expectation Sonification Model

This section presents the sonification model for movement expectancy developed from Huron’s ITPRA theory.

We also consider how the participant’s active role may alter their perception of the expectancy, and how the embodied link between body movement and music can be utilised to direct movement through sonification. Through this we develop different stages of the Movement Sonification Expectation Model (MoSEM), which explores how musical expectation within sonification impacts people’s movement perception and behaviours.

3.1.1 Stage one: expectation of movement

The goal of sonification is to represent the underlying data [20] and in the case of movement sonification, to extend the proprioceptive feedback. A purely informative sonification would improve the information available to the individual to make their prediction of expected movement and may help with the evaluation of whether it was correct. In practice, a purely informative sonification is difficult due to the musical and embodied way people tend to interpret sonifications [16, 31]. Nonetheless, this approach is considered here to examine how the information provided by sonification may impact one’s experience of movement.

Fig. 1
figure 1

Applying Huron’s theory of expectation to a purely informative sonification (used to track movement), showing the different stages that an individual experiences the sonification: During the sonification, the end point of the sonification and after the sonification. The imagined movement and the anticipation of the expected end are clearly defined. However, the movement itself and the appraisal of the movement remain unchanged

Full size image

Sonification aids people in tracking their performance of a movement, something that may be difficult when one relies purely on one’s proprioception: the sound can augment one’s understanding of the movement [1].

This can be seen in Fig. 1 where the expected endpoint is well-defined and the relationship between the imagined movement and the anticipation of the movement is clear. When considering populations with low self-efficacy, the expectation of the movement may not be as strong due to the limited expectation of their own abilities [32, 33]. This uncertainty leads to people being unsure of what they can achieve or how far they have moved. Low self-efficacy also makes it difficult to assess one’s completed movement [34]. In this case, if we consider the responses to the expectation created, the Imagination and Tension responses may be limited by one’s own belief in one’s own ability (low expectation of achievement). This, in turn, is reflected in the Prediction and Reaction responses in which this limited belief impacts the amount of movement performed, and finally appears in the Appraisal response where one’s ability is again assessed as limited, making difficult to progress [35].

3.1.2 Stage two: music to enhance movement expectation

Consider a musical sonification in which, as the individual moves they receive feedback in the form of progression through a musical phrase (e.g. as seen in [14, 17, 18, 29]): the expectation of the movement is enhanced. With the addition of this musical aspect to the sonification, the expectation in the movement is altered. As the sonification is generated from the movement and the individual feels agency over the sound, it builds a musical expectation, on top of, but related to the expectation of the movement (see Fig. 1). In addition to the feedback from the sonification of the individual’s movement, the design of the sonification causes an affective and emotional response [36]. The combination of one’s expectation of the music and the expectation of one’s own movement then impacts the way movement itself is experienced, with the two reflecting on each other. Due to the agency felt over the sonification and its use to enhance the proprioception of the individual, the musical aspects of the sound become linked to one’s perception of one’s own movement.

Fig. 2
figure 2

Matching musical expectation with movement expectation. The movement begins and tension starts to build to an expected ending; musical expectation is created by the sonification. At the endpoint of the movement, the combination of movement and music is reflected in the person’s reaction/prediction responses. At the conclusion of the movement, as the musical expectation matches the expected movement, there is an attribution of the musical feeling of completeness to the movement itself. Then as the music is appraised as complete, the completeness of music is also felt towards the movement

Full size image

Considering Huron’s ITPRA responses, firstly, the Imagination response begins to build the individual’s expectation of the music and by linking this to the ongoing movement used to trigger the sonification, entangles the two. From this, the Tension response develops as the tension in the music builds to this expected point; this is reflected in the continuation of the movement to the expected stable outcome. As the musical expectation and movement expectation match, when the endpoint is reached the Prediction response gives a positive reinforcement that a correct prediction of the music’s completion is found and thus the Reaction response shows this reflected in the movements at the target point to begin the return, as the movement, in turn, is perceived as complete. From this the Appraisal response assesses the correct prediction of the movement completion, and because the musical expectation is also met, there is a sense of reward from the completed music. This can be seen in Fig. 2 where the context is established, an expectation built and then met, leading to a feeling of completion and the conclusion of the movement. This is seen in the combination of the expected movement and the expected musical endpoint being aligned creating an actual endpoint of the movement in which there is a feeling of musical resolution.

In addition, this builds on Huron’s discussion of the “prediction effect”: a by-product of the misattribution of the positive feeling that comes from a correct prediction of the outcome of the music to the music itself. In this case, the MoSEM shows how this prediction effect may impact one’s perception of the movement which caused the sound, leading to a feeling of reward for the “complete” movement.

This has the benefit of announcing the movement and providing attributional feedback [34], through a sense of completion created by the sound. The emotional connection to the music created [37], and its connection to one’s movement helps support one’s ability to self-attribute the feeling of success, which is needed to overcome low self-efficacy. This supports the claim that sonification can be designed to promote positive affect [36] and that musical structure can be used to enhance sonification’s impact more generally [38].

3.1.3 Stage three: music to adapt movement expectation

Now consider a musical sonification that seeks not to match one’s expectation of the movement, but to alter it. This idea utilises the idea of the commonly-experienced feeling that musical tension needs to “move” to resolution (see the discussion in Sect. 2 on using auditory feedback to alter perception).

There is a link between the idea of musical expectation and movement felt in the embodied experience of music, and through leveraging the conceptual metaphor linked to the continuation or resolution of music to one’s movement, one’s perception and hence behaviour may be altered, as seen in Fig. 3.

Fig. 3
figure 3

By altering the musical expectation (shown in blue/red) felt through the sonification, the movement expectation is altered in turn. As in Fig. 2, the movement begins, and tension starts to build to an expected ending. At the same time a musical expectation is created by the sonification. However, in this instance, the conclusion of the music does not match the expected end of the movement. Due to the interconnectedness between the individual’s movement and the sonification, this in turn leads to the movement and appraisal of the movement being adapted to match the musical expectation

Full size image

For example, Chew et al.’s work explores cadential tipping points [39] where a point of musical tension is likened to “a physical object balanced at its tipping point, like a train atop the hill of a roller coaster, before it falls back into motion”. Huron also presents a Qualia survey [22], in which responses from people trained in western classical music who described different scale degrees, i.e. intervals between two musical notes in a major key, were analysed. While the tonic was described as stable were described as “home” and “contentment”, scale degrees which were perceived as musically unstable were described with words “pointing” or “restless”. These descriptors imply an association with musical instability and the desire for on-going movement, while conversely, musically stable sounds invite a feeling of finality and pleasure. Through associating these musical aspects with one’s movement, sonification can be designed to evoke certain responses from the user.

Huron’s theory of expectation considers one singular ITPRA response, however, there are likely to be many ITPRA responses taking place in response to the sensory feedback given, as the individual takes action to continue their movement based on the feedback. Having not reached the perceived endpoint of the movement, the reaction is to move more. It is through this ongoing building of expectation that the movement and music become intertwined as the continued movement continues to generate sound. If the expected musical endpoint is altered not to align with the expected endpoint of the movement (for instance through harmonic instability), the actual endpoint of the movement is altered by the desire for musical resolution because the sound is also acting as an extension of the proprioceptive feedback.

In more detail, where the music is left unresolved at the expected endpoint the musical expectation signals to the individual that the movement should continue. The tension felt by the individual delays the Prediction response, Reaction response and Appraisal response because the music has yet to end as expected, thus the individual continues moving towards some expected resolution. This leads to a drive to continue moving and an extended movement. However, if the musical expectation is resolved before the expected endpoint of the movement (the Prediction response that the music has reached its expected conclusion) then the Reaction response creates the feeling that the end of the movement has been reached. From this the Appraisal response assesses the correct prediction of the musical completion, which again is reflected in the perceptions of the movement being complete.

This builds the final part of MoSEM, from which some predictions on the impact of MuES on movement behaviour and perception can be made:

  1. 1.

    Should the sonification end with musical stability, the movement will be concluded and will elicit a feeling of completion and reward.

  2. 2.

    Should the sonification end with musical instability, the movement will continue toward the implied resolution and will elicit a feeling of motivation to continue past the endpoint of the movement.

4 Investigating musical expectancy sonification and other perceptual cues

This section describes a study to explore how musical expectancy-based sonifications impact people’s movement behaviour and is used to evaluate and extend the MoSEM presented above. While previous works have explored how musical expectation may be used within movement sonification [4, 17, 30], it has yet to be examined in relation to how one’s expectation of one’s own movement is impacted by it.

For the purposes of the study, we consider a single movement, raising and lowering an arm, and define “open” and “closed” versions of it. This movement was selected since, in the absence of non-proprioceptive cues, it is inherently open: there is no particular start/end point. We define a closed version of this movement by providing visual cues that bound the space within which movement should happen (see Fig. 4); the visual cues do not themselves define the actual extent of the movement itself, merely visually structure the space in which it can take place.

People are primarily “visually dominant”, i.e. they will rely more on visual input than other modalities [40]. Petrini et al. show that children will even use irrelevant visual cues to help localise sound, showing that learning to ignore these cues can be difficult [41]. This well-known dominance of visual cues over other types [40] means that we effectively create a perceptual closure on the arm movement without altering the proprioceptive cues an individual receives (because the fundamental movement is unaltered, merely the cues to its extent). This allows us to set aside specific considerations of proprioception and focus on the effects of the sonification in the context of the two movement types.

Visual cues (marks) were indicated on the wall in front of the participant for closed movement scenarios, and participants were asked to close their eyes to remove the visual reference for open movement scenarios (see Fig. 4). The aim was to test the hypothesis that the perceptual cues available to an individual at the target point of a movement change the way the musical stability in the sonifications impacts their movement. We adopt a musical expectancy-based design for the sonification (like that used in [4, 30]) and measure the impact the sound has on movement in the presence or otherwise of visual cues. A video demo of the sonification can be found [link removed for review—see additional files]

Fig. 4
figure 4

The two silhouettes show our experimental set-up using a simple arm movement: participants move within the calibrated space with the visual markers in front of them (with eyes either open or closed). The musical score shows the full sonification with either a musically finished (ST—stable) or unfinished (UN—unstable) ending

Full size image

4.1 Participants

A total of 23 healthy paid participants were recruited for the study (age = 18–57 (mean = 27.82), 13 female and 10 male). All participants reported normal hearing and vision. The mean score for the General Musical Sophistication subscale of the Goldsmith musical sophistication index was 69.34 (max 114 and min 43), with 19 out of the 23 participants were found to be below the mean score the general population (81.58) [42], therefore we assume the effects seen are not due to musical training.

4.2 Materials

The experiment used a smartphone-based set-up; a smartphone was placed on the top of the arm and the phone’s 3 axis gyroscope (~50FPS) to the z-axis angle was used to calculate the movement. Participants were positioned at a distance away from the wall so that the visual markers were in front of them, with their arm straight so that they could move their arm within the space regardless of height. The system was then calibrated in the space between the visual indicators, without reaching them, so participants were not moving exactly to the extremes of the visual indicator, using it more as a cue than as the target point itself. As the arm moves through the space a series of chords are played from the smartphone, ending in either one that meets musical expectancy (stable) or one that defies it (unstable) (see Fig. 4). The participants would start with their arm raised and then begin moving their arm downward, triggering the chord sequence as they do so down toward the end cadence, the sequence would then be reversed as they raised their arm again.

4.3 Design

A randomised within-subjects design was used for the four trials. There were two independent variables: harmonic stability (stable/unstable) and movement type (open/closed). Stable conditions had a musical resolution at the target point, whereas unstable conditions were musically unresolved. As described above, participants were either given additional visual cues (visual indicators), creating our closed movement, or no additional cues (eyes closed), creating our open movement.

To understand the impact that the different perceptual factors had on the way participants moved at the target point, three measures were taken from the data:

  1. 1.

    the amount of movement made past the target point,

  2. 2.

    the time taken to return to the target point, and

  3. 3.

    the speed taken to reach the maximum extension.

The self-reported measures were: perceived stability of the music the end of the movement, perceived confidence that the participant had reached the target point, perceived informativeness of the sound, perceived motivation to continue past the target point and perceived amount of reward. These were all measured using a seven-point Likert scale taken after each condition in which participants were asked to rate their agreement with statements for each measure (1 Not very–7 Very), for example, “at the end of the movement I felt rewarded”. Stability was defined as in [26], with 1 being completely finished and 7 being definitely needing to continue on to resolution. The self-report measures can be found within the supplementary materials for this paper. In addition, participants were asked to give a series of qualitative responses after each trial, on how they felt they used the sound, how easy the task was and how they felt they had performed differently from previous trials.

4.4 Procedure

After an initial introduction and demonstration of the device, the smartphone was placed on the participant’s arm using a phone holder. Participants were instructed to move their arm to the set target point and back to their starting position for four repetitions using the music produced to guide them for the four conditions. Specifically, they were told that when they felt they had reached the end of the chord progression, (i.e. the target point at the end of the movement space) they should return. No other instructions were provided. Each condition was presented in a randomised order and the sessions lasted approximately 30 min.

4.5 Results

Fig. 5
figure 5

Mean (SE) amount of movement, time of return and speed of movement for stable and unstable cadences in both open movements, no additional cues, and closed movement, with additional cues. From this the impact of both the stability and additional cues on the expectation of the movement can be seen

Full size image
Table 1 Median (range) values for self-reported data for stable and unstable cadences for both open and closed movements
Full size table

Firstly, behavioural measures were averaged over repetitions, and both behavioural and self-report data were averaged across lengths, to focus on the key variables in this study: the stability and the movement type. Descriptive statistics for all measures are summarised in Fig. 5 and Table 1, which show how the different cadences impact the movement behaviour and perception, in both open and closed movements.

We analyse the results using Anderson’s Information Integration Theory (IIT) that can be used to explore how different external stimuli are integrated into peoples’ perception to elicit a given response [43]. By establishing the rules by which people integrate information sources through simple algebraic formulations (defined as cognitive algebra), Anderson shows how we can understand the way in which an individual uses external stimuli in conjunction with each other to establish an internal understanding and perception. The main models considered in IIT are the additive model (where there is an impact from both factors), the multiplicative model (where one factor multiplies the impact of the other) or the averaging model (where the two factors are averaged together to build the individual’s perception). While Anderson’s work focuses on how this can be used to understand scale ratings of perceptions, recent work has shown how physical movements can be used similarly, including how different cues impact responses in movement [44].

Fig. 6
figure 6

Factorial plot and functional measurement marginal mean values for the amount of movement

Full size image

In our case, we can use IIT to explore how the musical stability and visual cues integrated to guide the ending of a movement. Through this method, we can better understand how the different factors contribute to the changes in people’s movement and explore the mental model people build for differing stabilities when provided with visual cues. By investigating these factors through the lens of IIT, we can better understand how movement-altering sonifications can be applied to different movement scenarios.

Firstly, a visual analysis of the factors is done to establish the appropriate type of integration model (additive, multiplicative, averaging). This is then validated by a repeated-measures analysis of variance (within ANOVA) with stability and visual cues as within-subject factors. From this, we can then examine the relationship between the two factors through a functional measurement of the integration, which in this case can be taken using the marginal means for each factor: marginal mean for stable endings = (mean for stable endings for the movements + mean stable ending for closed movements)/2.

This was done for all three of our behavioural measures: the amount of movement, the time taken to return, and the speed of the movement.

4.5.1 Amount of movement

Visual analysis of the factorial plot for stability and movement type (Fig. 6) for the amount of movement, shows parallelism between the conditions (i.e. the lines connecting the two points for the stable and unstable conditions are parallel), consistent with an additive model for integration, i.e. both factors have an impact on the way people continue moving as they perceive the end of the movement space.

The functional measurement for the amount of movement (also shown in Fig. 6) shows an increase in the amount of movement both in closed movements and when having an unstable ending. This is reflected in the repeated measure ANOVA showing a significant effect of stability (\(F(1,22) = 4.713, p =.041, \chi ^2 = .176\)), while the effect of movement type (open vs closed) was found to be approaching significance (\(F(1,22) = 3.541, p =.073, \chi ^2 = .126\)). In addition, in support of the addition model of integration emerging from the graph, there was no significant interaction found with the amount of stretch. Conclusion: Participants stretched longer in unstable conditions, and slightly further when the movement was closed (using an additive model of integration).

4.5.2 Time taken

Similarly, a visual analysis was done for the plot (shown in Fig. 7) for the time taken to return to the endpoint of the music. This time the diverging nature of the plot (characterized as a fan shape of the lines connecting the points for the stable and unstable conditions) suggests a multiplicative model of integration, where the movement type amplifies the effect of the stability.

Fig. 7
figure 7

Factorial plot and functional measurement marginal mean values for the time taken to return (reverse the movement)

Full size image

The functional measurement (also shown in Fig. 7) for the time taken shows an increase in the time taken to invert the movement due the movement being open and having an unstable ending. Again, the outcomes from the ANOVA support this model. The impact of stability on the time to return to the end-point of the music was found to be significant (\(F(1,21) = 7.628, p = 0.012, \chi ^2 = .266\)), as was the effect of movement type (\(F(1,21) = 15.969, p = 0.01, \chi ^2 = .432\)) with no significant interaction. These findings, along with the fanning shape seen in the factorial plot, suggest that the impact of the unstable ending is increased in the open movements.Conclusion: People took longer in open movements and when the music sounded unfinished (using a multiplicative model of integration).

4.5.3 Speed of the movement

Finally, visual analysis of the speed of movement plot, shown in Fig. 8, indicates that while parallelism is present, there is seemingly no change in speed due to stability. This suggests that the movement type was the predominant factor for determining the speed of the movement.

Fig. 8
figure 8

Factorial plot and functional measurement marginal mean values for the speed of the movement

Full size image

The functional measurement for the speed of the movement shows an increase in the speed due to having a closed movement but there was no real impact of the ending, suggesting that only the movement type had an impact on the speed of the movement. The impact of movement type on speed between the end-point of the music and the maximum was found to be significant (\(F(1,22) = 10.186, p =.004, \chi ^2 = .316\)), However, no significant effect of stability was found nor an interaction between them, implying that only the movement type was used. Conclusion: Participants were faster when the movement was closed.

4.5.4 Self-report data

Self-reported data were analysed with Friedman tests, followed by Bonferroni-corrected Wilcoxon signed rank tests. A series of Friedman tests found significant differences for: perceived stability (\( \chi ^2 (3) = 22.353, p < .001\)), confidence (\( \chi ^2 (3) = 21.187, p < .001\)), perceived informativeness (\( \chi ^2 (3) = 29.638, p < .001\)), perceived motivation (\( \chi ^2 (3) = 8.912, p = .030\)), and reward (\( \chi ^2 (3) = 10.064, p = .018\)). Subsequent Wilcoxon tests (\(\alpha = 0.0125\)) showed that stable conditions were considered more stable than unstable conditions both in open movements (\(Z = -3.571, p < .002\)) and closed movements (\(Z = -3.631, p < .001\)).

For confidence on reaching the target point, the tests showed that participants felt more confident in stable conditions than unstable conditions both in closed (\(Z = -3.627, p < .001\)) and open movements (\(Z = -2.921, p = .003\)).

For informativeness, the tests showed that participants felt more informed in stable conditions than unstable conditions both in open movements (\(Z = -3.441, p = .001\)) and closed (\(Z = -3.211, p = .001\)).

For motivation, results approached significance for stable versus unstable conditions in open movements (\(Z = -2.220, p = .026\)) and closed movements (\(Z = -2.077, p = .038\)), with participants more motivated to go past the target point in unstable conditions. Additionally, they approached significance when comparing unstable conditions in closed and open movements (\(Z = -2.184, p = .029\)), with participants feeling greater motivation in open movements.

Participants found stable conditions more rewarding than unstable conditions in closed movements (\(Z = -3.152, p = .002\)).

4.5.5 Qualitative analysis

The qualitative answers participants gave on how the different sounds/movement types affected their movements were recorded on paper during the discussion of the different sounds. They were then digitised and coded in F4 analyse to explore how participants used the sonification. The responses showed that 13 participants felt they relied on the sound and were “more focused” on the sound in the open movement, while 14 participants reported that they either relied on or were distracted by visual cues (the markers or indeed their own arm) in the closed movement. Additionally, eight participants reported feeling more confident in their movement in the closed movement as they could see they had reached the end of the movement.

Fig. 9
figure 9

The figure shows how external (non-sonic) cues can impact the use of musical expectancy as an endpoint predictor in a closed movement; the additional cues to the movement’s ending impact the degree to which the movement can be altered by the sonification. While this impacts the initial reaction to the sonification, in terms of movement, it is still appraised with the musical expectation

Full size image

5 Discussion

From the analysis of these results, we can see how the execution of different movement types and the reflection emerging from their execution (confidence, accomplishment) are impacted by an expectancy-based sonification. Firstly, it can be seen that when musical expectation is defied, participants move more and for longer than when it is met; this reflects the Reaction response to the music stimuli, to either complete or continue the movement. In terms of Prediction response, we see that people’s rating of stability holds with and without visual cues, with people’s musical expectation being met. Finally, self-report data show how the different stabilities impact people’s Appraisal response, with people feeling more confident, informed and rewarded in stable conditions. These results help evaluate the predictions presented in the MoSEM and demonstrate how changes in the musical expectation created can alter people’s movement behaviour and perceptions.

However, these results also demonstrate the impact of external cues on the effects of the MoSEM. Participants moved more in closed movements and they took longer to return in the open movements. Additionally, the qualitative measures demonstrate that while participants were able to concentrate on the sound more in the open movement, in the closed movement they were able to use visual cues to help guide their movement (though some deemed it a distraction). These results show how open and closed movements may interact with musically-informed sonifications.

Our analysis shows that the change in stability and movement-type both impact movement. Interestingly, participants moved more when in the closed movement, suggesting that people were still using the visual markers as strong cues as to where they should move before they are impacted by the stability (reflected in the additive model of integration). Similarly, as shown in the self-report, participants felt more rewarded in stable conditions only in the closed movement, perhaps again, as shown in the qualitative results, they are able to use the visual markers and their own arm as a confirmation that they have completed the movement.

On the other hand, the increased time taken for open movements shows uncertainty in the movement when endings were unstable. The suitability of the multiplicative model for the time taken can be seen as participants being unable to use the visual feedback from their arm for confirmation when their eyes are closed. This is also demonstrated in the greater impact of stability on motivation to continue one’s movement in an open movement over a closed one, with participants more strongly encouraged when there are no overriding cues from the environment. Again, this is reflected in the qualitative findings that they were better able to focus on the sound in the open movement without the visual distractors.

Comparing these results to previous works, we can see how stability can be used to alter movement [4]. However, some impact is seen on the time taken which correlates with both people’s reliance on visual cues [40] and faster response times for musically complete stimuli [45]. Participants moved less in the stable endings compared to unstable, and as suggested by Dyer et al., the design of these musical sonifications could improve the learning of movement patterns [5]. While these results and the presented literature indicate that there may be an impact of the movement type on the way musically-informed sonification alters movement, further study is needed to investigate, by limiting the use of proprioceptive cues in order to evaluate the kinds of movement behaviour-altering sonifications can be used with.

5.1 Movement Expectation Sonification Model

These results can be used to expand our Movement Expectation Sonification Model (MoSEM) to see how these external cues may interfere with our movement expectancy. As seen in Fig. 9, the addition of visual or other cues external to the sonification, plus proprioceptive feedback, can be thought of as an additional cue that the end of the sonification has been reached, hence the change in reaction response.

As shown in the original MoSEM, by making the person an active agent in the expectancy, the perceived event onset of the ITPRA response can be manipulated through continued musical tension. The results of this study show how, when other kinds of cue are available, they will impact people’s perceived expectancy. Moreover, the dominance of visual cues over auditory ones will lead to these cues (if present) being used to establish the expected endpoint of the movement and thus create a concrete event onset at which a person will form their prediction and reaction responses.

5.2 Implications for design and future work

The MoSEM presented here gives a model to design and understand movement-altering sonifications. In this work, we show how these kinds of sonifications may alter someone’s perception of their own movement, as well as a consideration for the suitability of different movement types. While the main goal of sonification may be to convey information through sound [20], this work demonstrates a need to consider how people’s perception of sound may alter how they use the sonification. When considering musical sonifications, the prior discussion in the field has focused on defining boundaries between music and sonification [20], the use of musical sound within sonification to improve aesthetics [16] and creating more effective conceptual metaphors [31]. Vickers et al., highlight the potential for the use of musical structures in sonification to disrupt the representation of the underlying data [16]. However, this MoSEM offers not only a lens to better understand how musical sound may alter this representation of data, but also shows how this can be leveraged to alter perceptions of one’s own movement deliberately.

This work demonstrates the need to consider people’s perception of sound (and music) in the design of sonifications and the need to consider non-sonification based cues. It also highlights musical expectation as a useful tool for use in sonification, which can be used to alter people’s perception of movement, useful in exercise [30], rehabilitation [4, 17] and motor learning [5], and through which sonifications can be designed not only to inform but to encourage as well. Future work in this area should explore how the MoSEM may be applied to these different areas, and how in turn the MoSEM can be expanded to consider not only extra-modal cues but perhaps also internal barriers such as self-belief or confidence in a movement [14] and how the MoSEM itself can be used to design effective sonifications.

6 Conclusions

In conclusion, this work shows that this kind of sonification may be dependent on movement type. The results demonstrate how the context of the movement must be considered when designing movement sonification, and the impact of other sensorial modalities must be explored when these sonifications aim to alter people’s movement behaviour. Similarly, while this instance of movement expectancy is built with musical expectancy in mind, similar thinking could be applied to other such movement-altering sonifications to explain how the way people expect a certain movement to proceed can be impacted by sound (as in [2, 3]). This paper has explored how musical expectancy can be used within sonification to alter one’s ‘movement-expectancy’. Additionally, it shows how movement type may affect how musically-informed sonifications impact movement. The model presented here helps to provide understanding of how musical structure can be used within sonification design to alter movement behaviour and perceptions.