Keywords

1 Introduction

The ability to enhance recovery of motor and cognitive function following brain injury remains a significant challenge for rehabilitation professionals. One of the greatest impediments to recovery is the patient’s reduced capacity to engage in therapy and to persist with it in the face of significant impairment [1]. People with an acquired brain injury (ABI) caused mainly from stroke or traumatic brain injury (TBI), often experience a range of physical, cognitive, and psychological issues that can have long-lasting and devastating consequences, both for the sufferer and their families. The psychological problems including isolation, low self-esteem, reduced social support, apathy, chronic anxiety and depression, are often part and parcel of the psychosocial deficits experienced among people with neurological impairments [2, 3]. This presupposes the need to design solutions in therapy that encourage social interaction and a more playful orientation to participation in rehabilitation.

Group-based rehabilitation that is playful in nature, thematically appropriate, and engaging has been shown to enhance rehabilitation outcomes. For example, the work of Green and colleagues have designed a program to encourage children with hemiplegia to learn (bimanual) magic tricks involving assorted props (like cards), culminating in a magic show after a 3-week program [4]. This program has demonstrated significant improvements in motor function and, based on child and parent reports, improved interaction with typically developed peers and family members at home. Another recent group-based approach with hemiplegic children is the Pirates program of Aarts and colleagues [5]. This program is based on the fundamental premise that play is the primary means by which children interact, learn and derive meaning in their world. Here groups of children dress as pirates and are encouraged to role-play scenarios that involve bimanual and locomotor activity using a variety of props such as plastic swords, treasure chests, “booty”, etc.

Both these programs demonstrate that playful interaction, role-play and performing with others provide opportunities for peer modeling, cooperation and friendly competition, all of which provide a fulcrum for goal-related activity and practice. Indeed, the rehabilitative effect of these group activities is regarded almost as an incidental outcome of the play itself. The playful and social context of these programs are inherently motivating, an essential aspect of therapy efficacy that is often neglected in the design of new rehabilitation systems [6].

Rehabilitation computer systems and custom off the shelf games (e.g. Nintendo Wii) commonly direct participants to focus attention on a wall-mounted vertical screen or head-mounted display but generally not on other participants [7]. These configurations can hinder social interaction and reduces opportunity for more complex interpersonal communication and vicarious learning [8].

Alternatively, with large format interactive walls and tabletop displays comes the possibility for computers to facilitate the work practices of small teams co-located around the same physical interface [9, 10]. In particular, tabletop displays can support face-to-face multiuser interaction through the physical affordances of the horizontal interface or the users’ mental models of working around traditional tables [11]. The technology is able to support a broad range of user interactions such as multi-finger touch, hand gesture and simultaneous manipulation of physical objects [12, 13], has been shown to lead to collaborative learning in a group setting [14], and foster multimodal communication between healthy users [15] and clinical populations (such as children with Autism Spectrum Disorders [16]. How group work using tabletop technology can enhance motor rehabilitation of acquired brain injury is, however, relatively unexplored. Our research aims to design environments that may be exploited to enhance social aspects of group participation and physical affordances of tabletop displays.

2 The Resonance System

The Resonance system is designed to mediate small groups of people with a brain injury interacting with one another face-to-face, and enables an embodied, first person view of performance. We have created a suite of interactive environments that facilitate social interaction and group rehabilitation. Group participation is fundamental to the experience of Resonance and builds upon our prior work called ‘Elements’, which supports upper-limb rehabilitation for individuals with traumatic brain injury (TBI) patients [17–19]. Resonance utilizes a tabletop graphics display and four soft graspable objects used by the patients (i.e. tangible user interfaces, TUIs). We utilize the same tabletop platform and TUI’s for Resonance and Elements to greatly extend the range of therapeutic options available to individuals and groups of patients. Resonance provides unimanual and bimanual game-like tasks and exploratory creative environments of varying complexity geared toward reaching, grasping, lifting, moving and placing tangible user interfaces on a tabletop display. Each environment aims to encourage collaborative, cooperative, and competitive modes of interaction for small groups of co-located participants.

Our design approach draws from principles of Computer Supported Cooperative Work (CSCW), a sub-field of Human Computer Interaction (HCI), to understand how computers can mediate and support collaborative work in a group environment (see [20] for a detailed discussion). Tabletop interaction supports an embodied, first-person experience of user interaction, one that capitalizes on our physical skills and our familiarity with real-world objects, which reflects the ecological approach of rehabilitation [21]. The ability of tabletop displays to support awareness of others’ actions is often cited as one of the main benefits of collaborative face-to-face learning [14]. We considered how participants might observe, communicate, and learn from others, who are involved in playful activities [22]. Awareness of what others in a group are doing is essential in coordinating collaborative learning and achieving common goals around a shared activity. We considered a wide variety of collaboration styles, including working in parallel, working sequentially in tightly coupled activities, and working independently. We combine these ideas to design activities that support co-located participation and social engagement within motor rehabilitation contexts.

2.1 System Hardware

The Resonance system incorporates the latest advances in computing and visual tracking to maximize the level of user interaction. We are using a MultitactionTM 42” touch screen display with a fully integrated computer, multi-point touch screen, and marker-based tracking [23]. Multiple cameras are embedded in the unit below the LCD panel which sample at 200 Hz. The computer vision systems are able to see through the LCD panel using infrared light to identify multiple fingertips, finger orientation and complete hands. Physical objects can also be tagged with optical 2D markers. By placing the marker on the base surface of the TUI’s the computer vision system is able to identify each object and track its position and orientation when placed on the display. As an added advantage the system is compact, self-contained, and does not require calibration for ambient lighting conditions.

2.2 Tangible User Interface Design

A key feature of the Resonance tabletop display is its capacity to integrate and support the manipulation of physical objects such as TUIs in ways that are natural to the user’s body and their environment [24]. The development of naturalistic interfaces for user interaction is essential to optimize performance and improve access for patients with cognitive and motor impairments [25]. TUIs can exploit multiple human sensory channels otherwise neglected in conventional interfaces and can promote rich and dexterous interaction [26].

TUIs placed on the tabletop are the primary means for users to control features and events within the Resonance system. The shape and physical weight of each TUI offers the patient varying perceptual motor cues for physical action. For example, the TUI design assists patients to relearn movement skills akin to lifting a cup, tumbler, or similar-sized object, and refine/adjust control while moving it. These simple actions offer some elements of real world human experience through the ways one might manipulate real world objects.

Movement performance in brain injured patients is constrained by a number of physiological and biomechanical factors including the increase in muscle tone that occurs as a result of hypertonia, reduced muscle strength, and limited coordination of body movement [27]. In response, we have created three sets of Additively Manufactured (3D printed) plastic fused filament fabricated TUI’s of varying dimensions, with a soft flocked fiber covering (see Fig. 1). Participants with varying grasp width and strength can select the TUI most appropriate for their hand size and level of function. To provide additional feedback on object orientation, the TUI’s have a variable internal material density that relates to size, thus increasing weight distribution at the base, making them more stable and orientating them toward the table surface.

Fig. 1.
figure 1

A range of TUI’s of different size with a flocked fiber finish

Full size image

The form factor of the interfaces has also taken into account the deficits typically experienced by brain-injured patients, including perceptual difficulties in visual functions, perception of objects, impaired space and distance judgment, and difficulty with orientation [27]. Therefore, high contrast colors and simple graspable shapes were used in the present design of the TUI’s to assist a visually impaired user individuate each interface and ease cognitive load.

2.3 The Interactive Environments

We describe three environments called Synchroshapes, Resonance, and Paint Wars that offer co-operative, collaborative and competitive modes of playful user interactions, respectively. These environments have been developed using the Unity 3D game development software [28], and TUIO software protocol for object and touch event tracking [29]. An onscreen main menu enables the therapist to select up to four participants from a database of users, select challenge options for each environment, and specify which side of the display each participant is using. To begin each player must manually select a TUI they will use for each hand before commencing. Players register their selection in the system software by placing the TUI’s on a left and right hand graphic ‘base station’ next to their player name.

The Sychroshapes environment is a goal directed bimanual task for 2 participants who must co - operate with one another placing objects on targets to build a sequence of sounds. Players must attend to their target shape as well as the movements of the other player. To play, a pair of TUI’s must be placed simultaneously on a sequence of targets when displayed (see Fig. 2).

Fig. 2.
figure 2

A screen grab of the Sychroshapes environment. Participants must synchronously place the TUIs on the corresponding rectangular and triangular target. A colored circle that grows centrally from the target indicates successful placement. An animated visual entity that plays a looped musical sound is created for each successful attempt.

Full size image

The targets are surrounded by an on-screen barrier. Players take turns to lift and move their pair of TUI’s over the barrier and place them as accurately and simultaneously on the targets as possible. A visual link between the two targets highlights the level of synchrony between the placements of the TUI’s. A short looped musical sound is created and visually represented on the screen if the objects are placed simultaneously. The distance between the targets increases and the reaching range varies over trials. Subsequent sounds are created and added to the sequence on successful placement of the objects, resulting in a procedurally generated musical soundtrack. Failure to place objects synchronously produces no sound. As the level of challenge increases players must cooperate in the placement of the TUI’s. As the task progresses shapes from both players may appear on the screen at certain intervals. Both players must time the placement of TUI’s on their target at the same time to create a new sound effect.

The Resonance environment is a collaborative musical tool to compose sounds using the position and orientation of the TUI’s together with multi-touch. Here, groups of participants can mix and manipulate sound and colorful graphics in an aesthetically pleasing way (see Fig. 3). Complex soundscapes can be generated easily using a combination of simple gestures and movements. Each TUI produces a unique set of sounds we call notes, pulses and atmospheres. A number of colored lines emanate on a central axis from the base of each TUI when placed on the display. These lines visually subdivide the screen into colored polygonal fragments that produce an audiovisual note when touched.

Fig. 3.
figure 3

A participant using the Resonance environment to compose sounds

Full size image

Additional fragments (approximate range 3-25) are created as more TUI’s are placed on the display, enabling a larger range of notes that can be played concurrently. Single touch and multi-touch gestures generate softer and louder sounds respectively. Sliding the TUI’s across the display changes the size of the fragments. The pitch of the note changes based on the fragment’s position on the screen. Another feature includes a circular array of graphic buttons around the base of each TUI that generate glowing pulses at regular intervals when touch activated. These visual pulses travel outward along the lines emanating from the TUI and create a percussive sound when they cross an intersecting line. Moving the TUI so that the lines intersect at different points can vary the pitch of the sound. For example, intersecting lines nearer and further away from the base of the TUI raises and lowers the pitch of the pulse respectively. Finally, rapidly rotating the TUI controls the playback speed of an atmosphere sound (e.g. sound recordings of nature - wind, running water). Turning the TUI clockwise and anti-clockwise plays the atmosphere forward or backwards respectively. A circular ring around the base of the TUI represents this feature. The volume of the atmosphere can be adjusted using a pinch-like finger gesture over the ring. For example, pinching the ring inward reduces the volume. Using a combination of these simple gestures and movements described, participants can collaborate together easily to create complex soundscapes.

In Paint Wars players compete in a race to collect a number of shapes that match the color and shape of their TUI (see Fig. 4). The shapes are gathered into a corresponding colored circular home base assigned to each player in one of four locations around each edge of the display. Players shoot a set amount of colored paint from the base of their TUI by touching the TUI to activate the stream of paint. The shapes swim along and move toward the home base if the virtual paint color matches. Players can spread the paint using their fingers to create colored pathways for the shapes to swim. The paint moves and reacts to touch using a 2D fluid simulation that mixes and blends the colors in complex ways. More paint can be added to the scene by refilling the TUI at the players’ home base.

Fig. 4.
figure 4

Two participants playing Paint Wars

Full size image

Players are instructed to create enough paint of their color to attract and guide the shapes they are collecting. Moreover, to increase the level of difficulty a series of virtual ‘pests’ can be added to the game to disrupt the player collecting their shapes. The pests include Magnets that momentarily draw shapes toward them, Ink Drains that attract and consume nearby paint, Sprayers that shoot out inverse paint color as they move, Repulsors that push the shapes away, Whirlygigs that mix the paint around, and Foils that use the paint to seek out a corresponding home base and steal shapes collected by players. Memorizing the functions of these entities is not important, but rather players should be able to dynamically respond to their influence on the shapes.

3 Preliminary User Evaluation

The current pilot study aimed to characterize the Resonance experience of patients with ABI and to determine the suitability and usability of the Resonance for this population. A total of 4 patients with ABI (Mean age = 41.5 years, SD = 21.73 years, 2 females, 2 males) were recruited for the pilot study, including one patient with acute demyelinating encephalomyelitis, two patients with TBI and one with stroke. A summary of the clinical and demographic information of the patients can be found in Table 1. The ABI patients were tested at least 4 months post-injury when neurological recovery was stabilized, corresponding to the sub-acute (> 2 weeks) stages of injury. The patients were still participating in the sub-acute phase of rehabilitation. The mean interval between the injury and our pilot session was 15 months. The mean age at injury was 40 years (SD 21.56 years). Exclusion criteria were pre-existing central neurological disorders, and musculoskeletal injury. The study was carried out in accordance with the principles of the Declaration of Helsinki and approved by the local ethics committee. Consent was obtained from all subjects.

Table 1. Summary of demographic and injury characteristics for the ABI group
Full size table

Participants were instructed to train for three sessions per virtual environment (i.e. Synchroshapes, Resonance, and Paint Wars), with approximately 12 min per session. For this pilot study, we divided the participants into pairs (Group 1 and Group 2). After experiencing each virtual environment, the participants were asked to complete an experience questionnaire and rate the following factors: (1) Engagement/enjoyment (e.g. Did you enjoy the musical sounds and/or sound effects?); (2) Feeling of control/usability (e.g. Did you know what to do with the objects?); (3) Effectiveness/success (e.g. Did you learn new ways to improve your movement?); and (4) Interaction with the other patient (e.g., How well did the system encourage you to interact with others while playing?). At the end of the procedure the participants were asked to fill in the Intrinsic Motivation Inventory [30] and they were asked which of the three environments they found the most enjoyable. The entire experimental procedure took place during a 60-min session for each pair in a room located in the rehabilitation center.

4 Results of the Evaluation

Synchroshapes. Individuals in Group 1 (KI and RE) and Group 2 (IA and ER) were engaged and enjoyed the environment. Both groups were able to perform the task. Despite a range of impairments (such as hemiplegia) all individuals were able to hold the objects (even with their hemiplegic hand) and perform the placement of the objects on the targets. They understood the task instructions. For instance, participant ER, who presented with more severe cognitive defects than the other three, took longer to understand the timing requirements of the task. This highlights the need for flexibility in the introduction of the VEs, to allow participants to learn at their own pace. Competency was mixed – consistent with the injury and cognitive loading. The environment allowed a level of co-operation between participants. For example, Group 1 verbalized a countdown together as a strategy to synchronize their movements, whereas Group 2 used visual timing because one of the participants was non-verbal and less able to communicate.

Resonance. There was initial willingness and interest to engage with the environment. However, there was a rapid drop off in interest and motivation. We attribute this to the exploratory, user-driven, interaction style of Resonance potentially being counter-intuitive to the patients, in comparison to their typical rehabilitation.

In conventional rehabilitation, patients are accustomed to receiving instructions on which actions to perform, and then performing them. With the Resonance VE, participants may have been confused by the user-driven interaction style. In future, a brief discussion on how the Resonance VE operates as a rehabilitation tool will be provided to patients. Nevertheless, all participants rated their experience with Resonance positively, in terms of usability, and audio and visual design. In addition to the data on participants’ interaction with the system, we also considered the environmental constraints and influences on their experience. For instance, we noted that the ambient noise of the rehabilitation environment might have been distracting, as we were unable to exclude ambient noise from the training environment. This had a noticeable impact on the audio feedback quality in the Resonance environment and reduced their level of immersion.

Paint Wars. Group 1 competed well together and understood the task. They reported a sense of ‘losing track of time’, were immersed in the game’s interactivity, were enthusiastic about using all of the game’s features, and expressed a desire to play the game again. Participants KI and RE both rated their interaction with Paint Wars highly, in all areas.

Group 2 were not competitive in performing the task. Overall, they gave the impression of being two single-users operating on the same screen. Both participants had difficulty remembering the task parameters, and interface controls, and required constant prompting on how to interact with the environment. As stated above, the participants in Group 2 presented with greater cognitive and communicative impairment than KI, and RE. Thus, we have concluded that the design of Paint Wars may need to accommodate a more gradual learning curve. Alternatively, it may indicate a higher baseline for cognitive function required for successfully utilizing this environment. Assessing these possibilities will be part of the next stages of development and testing.

5 Conclusion

Overall our preliminary findings from the usability trial have produced positive results, which justifies its continued development and testing. Situating and testing the system in the actual working environment of the hospital was invaluable to our work. This allowed us to field test and gain insights into how patients used Resonance and how we may improve the system. We observed that each mode of play encouraged a range of social and physical interactions between groups of players. Players were able to observe others’ success in accomplishing certain tasks, which in turn may provide a sense of self-efficacy to the observer that they might also have the ability to accomplish the task. Importantly, the participants were able to use the Resonance system to coordinate multiple everyday upper-limb movements related to fine motor control and touch, moving and orientating their arm in space, and modulating the force with which they are manipulating an object. These are functional motor skills that ABI patients struggle to perform and are able to practice in a fun, motivating, and engaging way using the Resonance system.

Based on the information gathered in this usability study, the next stages for the Resonance system will be to refine the design and the administration procedures. We will expand our research to include larger sample outcome studies, including pre-post designs, trialing Resonance over several sessions, and assessing near and far transfer effects from the training.

Resonance provides a novel multi-user solution that has the potential to maximize patients’ potential to “learn from others”, to develop social skills and confidence, and to instill motivation to work harder with fellow patients. Our study indicates that tabletop rehabilitation activities that incorporate co-located social play afford a powerful therapeutic tool to engage individuals with brain injury socially in rehabilitation and motivate them to participate in therapy.