1 Introduction

A higher rate of decline in cognitive function in the elderly age group is being witnessed in populations worldwide [74]. Mild cognitive impairment (MCI) exists mostly in the elderly population (65+) and is characterised by the quality of cognitive functions not meeting expectations with respect to age and education level, however performance is effectively normal in daily activities [17]. In amnestic MCI (a-MCI) [50, 56], memory loss is predominant and there is a high risk of progression to Alzheimer’s disease (AD), whereas in non-amnestic MCI (na-MCI), other symptoms are prominent and there may be progression to other forms of dementia [21]. The pathological changes caused by AD are irreversible and contribute to both cognitive and physical decline [73] accounting for 60–80 % of dementia cases [5]. The interplay between cognitive and physical deteriorations affect an individual’s ability to perform basic activities of daily living (ADLs) which are progressively compromised over time, affecting balance, mobility and gait [4, 11, 20, 51] in late-stage dementia. Nevertheless, the deteriorations of cognitive function can be slowed or delayed if the symptoms appear at early stages, as with a diagnosis of MCI, and are given appropriate intervention, typically via a rehabilitation programme that includes a wide range of brain training targeted at various cognitive spheres. Digital cognitive training may potentially benefit those with cognitive dysfunctions more than traditional training due to enhanced motivation and engagement, which overcome the tediousness of traditional training [42]. Moreover, there is evidence that digital games develop the executive functioning of the brain and improve cognitive skills [41] with potential for mitigating issues caused by cognitive impairment, e.g. playing a 3D video game (Super Mario 64) has been found to increase the grey matter in an elderly player’s hippocampus [72]. Consequently, much research has attempted to promote game-based approaches to digital cognitive rehabilitation, which have attempted to incorporate gameful elements within the serious context of cognitive rehabilitation. For the elderly, much of this has endeavoured to use touch-based tablet devices to stimulate various cognitive functions while minimising cognitive load, although augmented, virtual and mixed realities are increasingly being used [44]. Additionally, research [e.g. 34] has shown that an involvement of appropriate physical interaction may support cognitive functions in the elderly, especially in the memory aspect. However, the vast majority of research on MCI rehabilitation has focused on exergaming interventions [76]. Approaches that combine both physical and cognitive training with a gameful approach are rare, and there are none that we are aware of that target the MCI group in this way.

Serious games may be characterised along a continuum, ranging from games for purpose through to experiential environments with minimal or no gaming characteristics for experience [39]. Serious games for rehabilitation may extend anywhere along this continuum by assisting in physical or cognitive functioning at home or in a healthcare setting, using various interaction technologies, 2D or 3D game interfaces, and game genres. They may be single or multiplayer, adaptable, and facilitate progress monitoring and performance feedback [55]. Consequently, we propose a serious game framework for MCI, MCI-GaTE (MCI-Game Therapy Experience), that integrates gamified physical interaction with cognitive rehabilitation for elderly players with MCI. The framework accommodates the range of the MCI player’s cognitive and physical abilities through a player profile, commonly-used therapeutic contexts and tasks, and notions of playfulness and gamefulness for facilitating motivation and engagement. In this way, serious games developed using MCI-GaTE may be used as effective rehabilitative tools to support those diagnosed with MCI. The framework also serves as comprehensive criteria against which a serious game may be evaluated to determine its pertinence and sufficiency for MCI players. The framework is derived from primary and secondary research processes to iterate themes for establishing a player profile and therapeutic, core gaming, and motivational elements within the framework, specifically: (i) research literature covering gameful cognitive and physical rehabilitation and playful experiences; (ii) resident profile data from a nursing home; and (iii) in-depth interviews with occupational therapists (OTs). Figure 1 depicts the adopted research approach. Sections 2–4 present results from each of these three research processes in turn. Then, we present the proposed framework derived from these results in Section 5. Use of the framework is illustrated through a prototypical immersive serious game, A-go!, in Section 6, which is subsequently evaluated with OTs to validate its effectiveness. Section 7 compares the framework with the state-of-the-art to confirm its relative contribution. Section 8 concludes the paper.

Fig. 1
figure 1

Research approach

Full size image

2 Related research work

This section reviews relevant research literature to identify suitable gaming, therapeutic and motivational elements within the framework. A thematic analysis was adopted to identify semantic and latent themes [12] by iteratively identifying fundamental gaming aspects that may be directly apt to physical and cognitive training to support an MCI player. Additionally, all serious game examples were classified and grouped into therapeutic tasks and scenarios which may serve as therapeutic elements in the proposed framework. It should be clarified that the purpose at this stage is theme identification for framework construction rather than determining when particular techniques would be best applied. Research is now reviewed under three related areas: gameful cognitive rehabilitation, gameful physical rehabilitation, and playful experiences.

2.1 Gameful cognitive rehabilitation

In cognitive rehabilitation (neurorehabilitation), the use of games for serious purposes is ubiquitous, but encouraging acceptance of rehabilitation among the elderly is lacking. Researchers have examined strategies for slowing down cognitive deterioration in patients and the use of digital therapeutic games may improve upon traditional rehabilitation, which is often tedious, and may increase long-term motivation in players to facilitate active participation. Traditional neuropsychological assessments are not easy to administer and carry out and thus it is increasingly common to promote serious games as a solution in healthcare, typically as a screening tool or an alternative method for neuropsychological evaluations to overcome clinical limitations [48, 65, 66]. MentalPlus® [48] is a digital game proposed to assess cognitive functions including attention, memory and executive functioning, based on various scenarios. For example, in the selective attention training scenario, the player must respond quickly to click one of six buttons labelled with various animal heads to trigger a flying bird to drop the selected animal into a moving cage. The virtual elements, including the flying bird as a carrier and animal heads as distractors, may work to motivate players to continue training where the players can discern their utility from the metaphorical approach. MentalPlus® was compared with Telephone Interview Cognitive Status (TICS), one of the diagnostic tools in testing cognitive domains, and found to be useful in players who have heart failure combined with cognitive impairment. Therefore, it can act as an alternative diagnostic tool for determining neuropsychological issues. Similarly, Episodix [66] assesses cognitive functions and differentiates the mental quality between MCI, AD and healthy individuals, in particular episodic memory quality, through the use of three game scenarios focusing on: immediate and short-term memory, semantic and procedural memory, and long-term memory with yes/no recognition functioning. Developed in Unity and running on a touchscreen-based device, it allows the elderly to better adapt to the game. Rather than being a digital artefact for clinical service, the gamified design elements (aesthetics, progression and scoring) can be seen as a supportive way for the players to improve the training flow and behaviours. The Preventive Neuro Health (PNH) system [42] is a gamified and personalised crowdsourcing-inspired tool for preventing cognitive decline in the elderly and those with early stage dementia. It consists of 42 cognitive exercises focusing on six cognitive functions (attention, memory, executive functions, orientation, gnosis and praxis) and can be used independently without specialist monitoring and initial neuropsychological diagnosis. To sustain motivation and improve performance, it adopts game design elements, such as metaphor to promote an unlocking feature for rewards based on earned points, and preferences and levels, to encourage users to participate. Undoubtedly, gameful features can maintain long-term player engagement in the cognitive programme and thus improve the cognitive performance of patients undertaking instrumental activities of daily living (IADL).

3D virtual environments (VEs) are common in neuropsychological rehabilitation [2, 22, 57, 67, 68] to provide high quality visualisations to the cognitively impaired which promote re-learning and reinforce IADL. Due to their nature, they can realistically emphasise ecological, economic, and social values. This realism supports patients in transferring the skills they have acquired in the virtual serious game to their real life. The 3D Neuropsychological Virtual Rehabilitation (NVR) simulation system [22] offers an accessible means for individual cognitive rehabilitation. It assigns neuropsychological tasks in a VE with strong gaming elements. For example, the game scene allows players to manipulate game objects and navigate the game space to complete cognitive tasks in first-person view. Scoring, time limit and feedback are provided throughout the game, while narrative-based messages and statuses in the game scene further facilitate involvement and realism. In a touchscreen-based 3D multitasking game for cognitive impaired patients to perform a virtual cooking IADL [67], point-and-drag interaction enables a player to perform specific actions, such as filling a pot with water, to imitate a real-life cooking scenario. However, gameful features are not supported since the cooking scene is simply a virtual representation of a real kitchen setup which the player explores without any specific goal. Similarly, the VIRTRA-EL web platform [57] aims at preserving cognitive capacity in the elderly, such as memory, attention, planning management and reasoning, through realistic rehabilitation. Game features are displayed statically, e.g. navigation arrows for camera movement within the game scene, a clock for orientation information, a map for location, and a virtual currency. These reflect real-life situations for the elderly to raise awareness of the context and fulfil the tasks step-by-step. The 3DVE reported in [68], focuses on stroke-based impairments. It supports two types of navigation skills, allocentric and egocentric, for stroke patients to orient themselves so as to improve their spatial cognition. In the 3D geometrical game scene, a map and distant objects are used to help patients orient themselves. Both types of navigation training are gamified with the aid of earning coins, and this serves as a means to determine the difficulty level.

Use of electroencephalogram (EEG)-based technology can help to produce more accurate results for evaluation. A serious game for Attention Deficit Hyperactivity Disorder (ADHD) and Attention Deficit Disorder (ADD) [2] focuses on improving the task attention span for such patients. By using an EEG-based device, brain signals from the patients can control the game objects and select action buttons via a brain-computer interface (BCI) through a wireless EMOTIV device. It is a single-level serious game with one goal, which requires the player to take as many yellow cubes as possible within the time limit using two EEG actions to manipulate the avatar in the game scene: a ‘push’ state will be activated when the player wants the avatar to go straight while a ‘neutral’ state will be activated when the player is still. The quality of the gaming elements in the game are controlled to achieve the optimal condition for the patients to attempt the tasks. For example, the humanoid avatar uses grey to reduce distraction, while the environment is mainly green with simple yellow cubes. The iPad-based serious game Decoder [59] is a cognitive game that aims to help visual sustained attention impaired individuals, who lack a sufficient amount of attention to accomplish tasks. The game assigns a Signal Intelligence officer as a role for the player to identify criminal locations by undertaking numerical tasks. These are adjusted in real-time for the player, and selection of character and backstory provides further personalisation. The game was found to have positive effects on the target group by promoting enjoyment and task-related motivation. Several significant gaming elements can be highlighted throughout the game for enhancing training effectiveness: the game uses distractors to hinder the player, and rewards via letters which serve as mission clues to lead them to continue the training.

In the tangible walking game [34], the functions of attention, memory and emotional quality in elderly players were tested. The use of physical activity, in this case walking, served to improve the quality of working memory, although visual attention and emotional quality did not show large variations. The first-person view avatar is a notable gameful feature that may support the walking simulation to achieve a realistic sense of control and achieve better performance during training.

In [10, 32, 36, 64], 2D serious games are proposed to support those with neuropsychological dysfunctions. In [10], different real-life daily routine scenarios are created as therapeutic tasks beneficial to those with cerebral dysfunction. The flat design simplifies the appearance of the game objects which lowers the level of visual recognition required. In [64], a touch-based whack-a-mole game is proposed for assisting OTs in evaluating a patient’s cognitive functions, focusing on hemi-spatial neglect through the use of a grey semi-transparent region over the playing area. After the training, a star rating for the patient is displayed and the OT is prompted to record notes. Game of Gifts Purchase [36] is a shopping game that uses characters with on-screen speech bubbles to explain the game scenarios to elderly MCI players. However, this may increase their cognitive load and cause them to lose interest in the game. Another study [32] attempts to improve prospective memory (PM) within the elderly by comparing traditional paper-and-pen tests with their digital versions in terms of the various elements provided by each. Digital training was found to beneficially provide realistic scenarios which contain distractors and model real-life situations, creating a sense of familiarity in users while undertaking the digital training. However, external cues in the environment, and the distractors which induce certain lifestyles and motivations, can affect performance in the PM tasks, while the level of attention in the task and its context can determine the efficacy of the PM training.

In the majority of the above approaches, 3D VEs tend to be favoured within the gaming scenarios to provide realistic adherence for cognitively impaired people. Compared with 2D VEs, the gaming specificities and project objectives in 3D VEs are relatively more suited to the target group. One study [14] has demonstrated that participants in 2D VEs and 3D VEs will carry different levels of cognitive load, with a 3D VE with sufficient visual cues offering advantages to participants with weak spatial ability. Cognitive rehabilitations that utilise a gameful approach may motivate MCI patients to attempt rehabilitation and the aforementioned examples mainly focus on cognitive functioning with light physical interaction for certain patients, one exception being the tangible walking game which combined both cognitive and physical tasks for those with neuropsychological issues. Thus, the extent to which physical tasks should be used to support the cognitively impaired person without creating excessive cognitive load needs to be considered. Physical rehabilitation is considered further in Section 2.2.

Table 1 summarises the above literature in terms of themes for the proposed framework that are subsequently used to derive underlying elements.

2.2 Gameful physical rehabilitation

Game-based physical rehabilitation, typically exergames, incorporates playful and gameful approaches for various physical rehabilitation contexts. Gamification of the training tasks provides gameful design components for patients to visualise game objects and engage with the game. Playful design helps to create joyful and motivational digital experiences. Gamefulness in physical games mostly employs gameful elements as motivators or distractors to facilitate engagement and attention of patients during the rehabilitation. As reviewed in this section, many studies utilise gameful elements to support physical rehabilitation in various contexts, such as oral palsy and diabetes, to facilitate improved engagement and motivation in patients through interaction with real-time game objects, shifting their attention away from the rehabilitation context. The gameful elements are designed according to the specific physical tasks the patients are required to fulfil.

mHealth [31] used a participatory design technique to identify various gameful design elements for a self-management and medical intervention app to help those with chronic physical illnesses: (i) points, progress, and rewards, (ii) goals, challenges and competition, (iii) avatars and feedback, (iv) social features, (v) themes, stories and narratives and (vi) engaging visuals, sounds and texts, for improving the overall user experience. Various studies [3, 9, 19, 24, 38, 45, 47, 60, 62, 75] examine gamefulness and the selection of gaming technology based on the target users’ individual limitations. For example, PhysioMate [38] helps wheelchair users by using a Microsoft Kinect sensor to motion track their upper-limb movements, such as motor functionality, to facilitate interaction with the exergame, while [60] uses VR-based gamified rehabilitation to help train upper limb movements through an avatar in the 3DVE which guides the patient in completing designated shoulder or arm movements and scores them accordingly. [19] exploits the advantages of natural user interfaces to develop a personalised game for patients with Parkinson’s Disease (PD), which trains their fine motor skills naturally. It considers PD’s physical capabilities, such as pinching, thumb opposition and grabbing hand movements, which are used to construct game scenes whose common gaming elements include scoring, graphical feedback, and a timer. It was evaluated according to immersion, flow, competence, positive affect, negative affect, tension and challenge, which serve as qualities leading to optimal motivation in players. The Escape game [3] for hand rehabilitation and Fun-Knee [75] for knee rehabilitation take advantage of Leap and wearable sensors. Both exergames are highly supported by gameful design elements, such as scoring, timers and rewards, to motivate patients to engage with the game scene and complete the rehabilitation tasks. Gameful elements may increase the effect of rehabilitation through visual cues. They act as a communicator for patients to recognise and experience the flow of the game to achieve the rehabilitation tasks. Flow [13] is a zone between boredom and worry that sustains the motivation and enjoyment in an experience. Fun-Knee is a successful example whereby the pain distraction element is incorporated as the distractor with the peak-end effect in order to reduce the pain and maintain the flow while patients are undergoing the Total Knee Replacement (TKR) rehabilitation process, a post-surgical knee therapy. This process utilises two inclinometers to calculate the knee angle of the patients while they are performing the heel-slide movement, where the input data is processed and displayed with gamified features. Pain is viewed as the foremost factor that affects the patient’s cognitive function. Using gameful design elements, such as the number of fishes for keeping score and the fishes in the tank as a reward, the patient can be distracted from the pain while they are catching the fish, which enhances their motivation and engagement. The gameful design elements reward the patients for their correct responses. Moreover, the promise of reward and scoring features are key to arousing the patient’s interest and increasing their motivation.

Table 1 Emergent themes for gameful cognitive rehabilitations
Full size table

Several exergames [9, 24, 45, 62] demonstrate that the combination of entertainment and therapeutic context can be highly beneficial to the quality of the rehabilitation process. They offer gameful environments for specific physical issues in order to motivate patients. In [9, 24, 47], participants are required to perform tasks to improve their balance and postural skills for reducing the risk of falls, as well as the functioning skills of human limbs for strengthening the foot muscles. These rehabilitation games use sports games (soccer and Pong) as themes for patients to complete tasks by performing certain movements and interacting with virtual objects. They require a wide range of body movements and offer goals which can be adjusted to suit the participants’ abilities. For example, in the HitIt fall prevention serious game [9], players move their head according to the soccer ball’s position, while in the PONG game [24], ankle exercises for human lower limb rehabilitation are supported by blocking the ball from touching the scoring region. The DoublePong 3D platform game [47] enables patients to undergo rehabilitation with AnkleBot, a rehabilitation mechanism, for training the lower body and ankle with the requirements of using vertical and horizontal movements. The user interfaces are comparatively simpler, utilising target game objects to lead the player to control small parts of their body precisely to navigate virtual scenes. These games attempt to produce more subtasks with different gameful elements. For example, a Kinect-based oral rehabilitation system [45] helps the patient undergo oral rehabilitation by requiring the mouth and tongue movements to interact with specific game objects, e.g. as virtual food is presented, the patient is asked to either bite or lick according to the type of food, so as to improve the score within a certain time frame.

Game features may also promote the entire rehabilitation package to target patients while simultaneously accruing ecological benefits. In ASPIRA [62], an embedded monitoring system checks indoor air quality to help asthmatic children, and the player is required to respond to prompts and alerts in order to achieve a healthy environment. The application adopts a “space” theme to help the player understand the procedure of the entire treatment process. It improves and encourages the player’s intrinsic motivation in order to develop their self-management through gameful elements, e.g. the personalised avatar, alert function and accumulated prizes or achievements can help to maintain regular engagement.

These exergames implement motivators and distractors which help patients engage and focus on the serious game, and can also act as tools to draw the attention of patients with cognitive impairment into memory training, which is crucial for completing MCI rehabilitation tasks.

Table 2 summarises the above literature in terms of themes for the proposed framework that are subsequently used to derive underlying elements.

2.3 Playful experiences

Play is a ubiquitous activity for all age groups [58] and playfulness refers to an attitude towards an activity in psychological, physical and emotional ways [61]. It can elicit emotion and intrinsic motivation in the absence of purposes and goals. Walz and Deterding [70] define gamification as “the use of game design elements in non-game contexts” and distinguish “gamefulness” from “playfulness” by the use of games and toys respectively. The ambient play circumstances can persuade or seduce people to explore, interact and enjoy experiences involving given playful elements [58, 63]. The elderly with MCI are likely to lack the self-motivation of engaging in rehabilitation actively and lack adequate technical knowledge [16]. MCI can lead to dementia, such as AD, which is correlated with motivational and emotional disorders including depression that will affect patient behaviour toward different activities [49]. Several studies [7, 15, 28, 30, 37, 40, 43, 58, 63, 71] provide a more specific indication of how playfulness affects people’s behaviour and psychology.

Table 2 Emergent themes for gameful physical rehabilitation
Full size table

In [30, 37, 40], inclusive playful experiences are introduced to a wide range of people, such as those with cognitive, physical and developmental impairments who behave differently, so as to improve their social well-being and advance their enjoyment while playing. In [40], the behaviour and appearance of the robot Iromec provides play scenarios to children with various medical conditions, such as cognitive impairment and motor impairment, who may not be able to play normally. The adaptive personalised experience encompasses a variety of movements and actions to manipulate the robot. For example, an emotionless Iromec embedded with a task for children to take turns interacting with the robot and a mimic scenario to allow them to follow the robot’s movements caters for autistic children, while coordination and sensory stimulation caters for those with mental impairment, and physical interactions with the robot cater for those with motor impairments. Similarly, Modular Interactive Tile [37] allows people of all ages to enjoy playful physical activities through moveable modular tiles with a touchable surface and coloured lights. Interacting with the tiles via the feet rather than the hands may also support falls prevention [30], one of the common serious problems in the elderly.

The player experience (PX) for the elderly in [71] applies skill balancing to playful elements, subject to manual and dynamic difficulty adjustment, facilitating intrinsic motivation, active behaviour and satisfaction. Manual adjustment was found to achieve a greater effect on the elderly than dynamic difficulty adjustment, and the elderly were motivated to challenge themselves through playing the game with playful elements that provided a high flexibility of free movement with the aid of Kinect in Safari Move, a zoo-themed game. Therefore, freedom of performance is important for eliciting active participation in players. Playfulness can also be utilised in collaborative situations. Multisensory Interactive Window [7] is a tangible interactive installation for the elderly to enjoy the digital experience at home individually by looking through a virtual window. The system targets those who spend a lot of time at home and are unmotivated to do outdoor activities. Thus, it enables them to re-connect their social life with those who are far away, removing geographical limits.

Playfulness can also potentially enhance the learning experience. PlayCubes [28] are dynamic objects for monitoring the constructional ability of children based on ActiveCube (AC), a tangible user interface (TUI). It provides a playful virtual circumstance to children to mimic playing in the playground. Similarly, eShadow [43] uses the concept of a traditional puppet show to attract people to harness their personal expression and creativity. The digital shadow theatre enables the player to participate and engage with the creative platform, so as to cultivate their social skills. It promotes active learning and an inclusive experience for children to enjoy.

Thus, playfulness introduces inclusiveness, a sense of freedom and potentially a collaborative experience for all. The mixture of fun and play promotes an explorative, social and enjoyable experience to target players who actively interact. Importantly, playful interactions have great potential to attract patients to participate in long-term physical and cognitive rehabilitation. By using playfulness as an attitude, active participation in cognitive rehabilitation is possible.

A list of themes derived from the playful experiences in the above literature is presented in Table 3.

3 Nursing home resident profiles

To provide a detailed understanding of the profile and background of elderly patients with MCI, and also uncover and refine further themes and elements for the framework from therapeutic settings, we analysed resident patient records that were documented first-hand by specialists and OTs in a nursing home in Hong Kong, GRACE Healthcare Ltd. All of the records sampled were from resident patients who had already been assessed as having some degree of MCI. In this way, the capabilities and needs of the MCI group in the serious game environment could be determined. In collecting and analysing the data, we aimed to: (i) construct an MCI player profile within the framework comprising background information, cognitive and physical capabilities in order to support adaptation of a therapeutic serious game; (ii) identify therapeutic elements and approaches in support of MCI players; and (iii) compare across sources to identify consistent themes for the serious game framework.

Table 3 Emergent themes for playful experience elements
Full size table

The participant records comprised diagnosis, duration of stay, results of various qualitative assessments that the residents have attempted under the supervision of specialists or OTs, and the basic background of each resident, i.e. education and social characteristics. These records are formed from analyses and summaries contained in assessment forms and individual care plans. We randomly sampled 31 current nursing home residents with dementia in the 72–100 age range (mean age = 86). All were able to participate in a set of relevant assessments and consisted of 23 females and 8 males. They were diagnosed and categorised into three stages of dementia: mild (n = 12), moderate (n = 18) and severe (n = 1). The diagnosis is made with evidence of memory impairment together with one cognitive domain deficit, which will affect daily life activity. Those with mild dementia and memory impairment serve as the core MCI subjects, with the moderate and severe residents serving to indicate and infer further participant capabilities and treatment for MCI. Across both sources, we identified five themes from the data which were used to develop those aspects of the serious game framework: (i) background information, (ii) cognitive capabilities, (iii) physical capabilities (upper limbs and lower limbs), (iv) therapeutic tasks and (v) therapeutic scenarios.

3.1 Assessment forms

The assessment forms indicated the general information for 26 of the 31 sampled residents, including their major cognitive and physical problems. Regarding background information, residents predominantly had no formal education, and only three with moderate stage dementia had completed primary school (Fig. 2). Almost all were either attentive or easily distracted, with only one resident unable to concentrate (Fig. 3). 23 residents were assessed as alert when conscious, while three were sleepy (Fig. 4). 69.2 % of all cases had mediocre tolerance levels and needed to rest during activities (Fig. 5). Regarding cognitive capabilities, seven residents (26.9 %) suffered from the five main cognitive impairment issues (attention, working memory, use of language, visuospatial skills, or executive functions), and, from these, three (42.9 %) were diagnosed with memory impairment. 53.8 % of those with cognitive problems participated in cognitive training, while 7.7 % of those with memory impairment took part in memory training. In respect of physical capabilities, almost all (92.3 %) suffered from some form of physical problems relating to the upper and lower limbs. 26.9 % of those with physical problems took part in physical training. Three physical assessments were involved: hand functions, functional tasks, and balancing. Hand functions and functional tasks required the residents to complete five tasks each using their upper limbs (Fig. 6). 96.2 % of all those participating managed to complete four hand functions (fist, opposition, pinching, and grasp and release), but only 69.2 % could complete the co-ordination hand function. However, all mild cases were able to complete the four hand functions. Balance was assessed using sitting and standing tasks for examining the lower limbs (Figs. 7, 8 and 9). As shown in Fig. 7, while sitting, with good condition, 53.8 % possessed static balance and 46.20 % dynamic balance, whereas while standing, with good condition, only 11.5 and 7.70 % possessed static and dynamic balance respectively. As can be seen, the cases in Fig. 9 are inversely proportional to the cases in Fig. 7.

Fig. 2
figure 2

Education level

Full size image
Fig. 3
figure 3

Attention level

Full size image
Fig. 4
figure 4

Consciousness level

Full size image
Fig. 5
figure 5

Tolerance level

Full size image
Fig. 6
figure 6

Physical (upper limbs) capabilities

Full size image
Fig. 7
figure 7

Physical (lower limbs) capabilities with good condition

Full size image
Fig. 8
figure 8

Physical (lower limbs) capabilities with fair condition

Full size image
Fig. 9
figure 9

Physical (lower limbs) capabilities with poor condition

Full size image

3.2 Individual care plans

The individual care plan is the detailed version of the assessment form. All 31 residents were fully recorded by their OT in terms of their cognitive and physical abilities, influencing factors, goals, and rehabilitation implementation. The main physical problems across all plans were recorded as: physical abilities/self-care, reduction of mobility, weak autonomous control ability, stiffness and contracture of limbs, and decreased amplitude of limb control ability. In terms of major cognitive problems, the cognitive abilities of most (26) residents were recorded as having deteriorated further due to ageing and extended hospitalisation, while 3 cases were due to being long-term bed ridden and a lack of social stimulation. Further to these, social and behaviour problems are influencing factors that should be considered when setting the goals for rehabilitation implementation. Common problems noted included restlessness, easily feeling annoyed, and being quiet or passive. Such problems may lead to a negative emotional state during activities. In order to improve the aforementioned physical and cognitive problems, six main rehabilitation implementations were recorded within the plans: rehabilitation exercises, cognitive training group, activities of daily living (ADLs), instrumental activities of daily living (IADLs), reminiscence activity, and reality orientation (RO). The latter four therapeutic scenarios present therapeutic tasks in various situations or settings, which would require the player to relate the assigned tasks to their daily lives, whereas the first two are general and non-targeted such as ad hoc hand functions or general training with other residents. Cognitive and physical capabilities must be taken into consideration for therapeutic tasks taking place within a scenario.

4 Interviews with OTs

To further understand the rehabilitative settings for MCI, and to consolidate the hitherto identified themes within the framework, we undertook interviews with four registered and experienced OTs from hospitals and nursing homes in Hong Kong. Given the anticipated difficulty of accessing OTs for the extensive periods of time necessary, we employed purposive sampling [69] whereby the OTs were selected for their ability to provide the rich detail and depth required for our analysis. A semi-structured method was used to allow for flexible and open-ended discussion [29], so as to directly address the problems with guiding questions as follows:

  1. Q1.

    To what extent do you agree that patients with MCI are interested in using computer-based rehabilitation/therapy? What makes them interested or uninterested (both initially and during the rehabilitation/therapy)?

  2. Q2.

    What can increase a patient’s attention and motivation during cognitive rehabilitation/therapy? What tools are used during the rehabilitation?

  3. Q3.

    In general, how long does an intervention usually take? How do you know it is working based on the patient’s capability? What is the minimum duration for it to be effective? Is there a maximum duration?

  4. Q4.

    What basic level of physical movements (especially arms/hands/fingers) are patients with MCI able to manage? Please give some examples.

  5. Q5.

    From the beginning to the end of the cognitive rehabilitation/therapy, which sessions or aspects are cognitively challenging for the patients? Please give some examples, e.g. attention, interactions, external/internal factors affecting them, etc. What factors might help make these sessions or aspects less challenging?

  6. Q6.

    From your experience and observations, do you think computer-based rehabilitation/therapy can potentially contribute to patients with MCI more effectively than the traditional training platform? How? In what areas?

  7. Q7.

    Can you show me some examples of cognitive rehabilitation with/without technological support? In your experience, what is the main difference between them? What are the positive and negative effects on the patients?

  8. Q8.

    What type of memory training have you used with patients (including currently) and why? What are the differences/benefits between them?

  9. Q9.

    Do you think patients with MCI are sufficiently competent to manage the basic functionality of a digital game? What are the common restrictions to be aware of? What range of interactions and range of physical movements do you think patients could manage? Using which parts of their body?

In this way, the interviews helped to further refine the MCI profile and the physical and cognitive therapeutic elements by harnessing their experience in designing and executing rehabilitations for their patients. Each OT was interviewed twice for 2–3 h on two separate days and further followed-up via email where clarification or further discussion were required. The interviews were conducted in Cantonese and audio recorded and transcribed into English prior to analysis. Analysis was undertaken using the thematic approach of Braun and Clarke [12]. Data were selected and transcribed through the data deduction process, with irrelevant responses and repeated responses removed. Given that potential themes were already identified from the literature and nursing home data, exploratory research was not the focus at this stage and thus interview responses not pertaining to the four core sectors of the serious game framework were not transcribed. To try and minimise such out-of-scope responses and to help the OTs understand the interview context, they were asked to review the initial construction of the framework in advance. Table 4 summarises the constructed themes from the responses to the questions. Participants are referred to as P1-P4 for anonymity purposes.

Table 4 Interview questions and emergent themes
Full size table

Eight of the themes (over 40 %) were mentioned by all OTs, in response to a single question or over multiple questions, suggesting their strongest relevance to the framework. These prominent themes were: attention support, duration, education, levels, physical functional tasks, reality orientation, reminiscence, and supervision & verbal encouragement. For example, for the supervision & verbal encouragement theme, responses included: “the relationship between the carer and patient is important for motivating them and building trust, because we understand their living styles and they would be willing to listen and follow our instructions” Q1-(P4), “assistants or carers can provide verbal encouragement… [and] sometimes, the participants would need us to play with them or just watch them do the tasks for a few minutes” Q2-(P1), “there is no verbal reinforcement provided by technological support … [so] the support of the OT can … motivate the patients to do the tasks” Q7-(P2), and “I’m sure that technology can provide accurate measurement, and it is good for data recording, but I don’t think that technology can replace the jobs of OT because we can provide concise analysis of our patients” Q7-(P3). Five further themes (ADLs, attention, autonomy: freedom of choice, IADLs, and tangible tools) were mentioned by all but one of the OTs. These themes reinforced those previously derived from the literature, such as tangible tools, or from the nursing home resident profiles, such as attention, e.g. “To initiate them into the rehab at the beginning would be the most challenging because it requires their attention and the correct mental state to perform the tasks. Sometimes, they will blame themselves and feel stigmatised…” Q5-(P4). An additional six themes were derived from more distinct responses. These suggested that what is potentially possible is not fully known to the OTs, such as personalisation: “It is hard to provide training that is tailored to each of them” Q2-(P1). In some cases, the theme seemed tacit due to its nature, such as coordination: “Co-ordination will always be involved, like using both hands to do the tasks, which mainly requires co-ordination and not just a simple physical movement. For example, picking up certain coloured objects with the co-ordination of the eyes and fingers.” Q4-(P2). In the next section, we integrate all the themes from the research results in Section 2 to 4 into a serious game framework that may be used to develop serious games targeted at players diagnosed with MCI.

5 MCI-GaTE: a proposed serious game framework

Drawing together the themes uncovered from related research literature, nursing home resident profiles, and OT interviews, we derive a serious game framework for MCI players, MCI-GaTE (MCI-Game Therapy Experience), presented in Fig. 10. The framework organises the themes as elements within four sectors: an MCI player profile, core gaming elements, therapeutic elements, and motivational elements. Related research led to a broad initial building of the framework, incorporating various therapeutic, core gaming and motivational elements. Most of these elements were further validated by the nursing home resident profiles and OT interviews. Similarly, analysis of the nursing home resident profiles led to the establishment of the MCI player profile elements, as well as additional therapeutic scenarios and core gaming elements, while the OT interview data established potential impairment elements within the player profile, and various motivational elements. The framework serves to conceptually represent the range of elements that may be incorporated when designing and developing a serious game for cognitive and physical rehabilitative support of patients with MCI. As such, it does not encompass particular implementation-level elements. Although the framework was influenced by the surveyed research which included solutions targeted at other diseases requiring rehabilitation, such as Parkinson’s Disease, and involving related serious games for health, such as exergames, the resultant framework is specifically focused on MCI rehabilitation only. Each of the four sectors of the framework is now discussed in turn and considered in relation to the results presented in the previous sections.

5.1 MCI player profile

The MCI player profile defines the range of capabilities of a player with MCI and supports the design of an appropriate serious game for them. The elements were derived predominantly from the nursing home resident profiles and were further refined by the OT interview responses.

5.1.1 Player background

From the nursing home resident profiles, three main background characteristics emerged: education, tolerance, and consciousness. These are foremost in reflecting the background of an MCI patient relevant to rehabilitation. While attention was also prominent, it is generally considered a main cognitive function, and therefore indicative of player cognitive abilities rather than background. As such it is considered in Section 5.1.3. The interviews reinforced that their degree of interest in using digital rehabilitation or therapy strongly depends on their education level Q1-(P1-P4). However, most of the nursing home residents were found to have no formal education (Fig. 2), which is to be expected given that they are aged 72–100 and historically there were few opportunities to have a formal education during their youth, but they may have higher practical skills related to their daily lives. This may affect their understanding and learning ability and therefore increase their likelihood of refusing or not continuing with a serious game. Despite this, 30–40 % of MCI patients are generally interested in performing therapeutic tasks using a digital platform Q1-(P1) and given that their cognitive impairment is at the onset, on-going explanation and assistance may help overcome any education issues.

Fig. 10
figure 10

The MCI-GaTE framework

Full size image

Therapeutic gaming tasks exhibit different contexts which notably require the tolerance capacity of the player to be sufficiently high to accommodate the requirements of the serious game environment. The majority of residents in the nursing home had a tolerance level where they required rest during activities (Fig. 5) and thus tolerance will influence what is considered a manageable period of time for the player to master a set of therapeutic tasks.

Consciousness is closely related to engagement during an activity and therefore important to take into account as well. The residents at the nursing home exhibiting mild dementia were all assessed as being alert and thus had sufficient awareness to respond to the surroundings. However, a small number of residents in the nursing home had a consciousness level of sleepy (Fig. 4), which would imply that they would be unable to manage any tasks.

5.1.2 Player physical capabilities

The OTs in the nursing home utilised several hand functions (opposition, pinching, fist, grasp and release, and co-ordination) commonly used in ADLs within the therapeutic training and testing of upper limb capabilities, particularly on both hands. For example, needling during sewing and picking up small items involves opposition and pinching actions that require control of the smaller or fine motor muscles of the hands; fist and grasp and release are operated for tasks such as wringing a towel or moving objects, which require muscle strength; and co-ordination is a popular action in rehabilitation where a series of movements switch the workload between left and right hands freely. Co-ordination may also involve hand-eye coordination to complete tasks, such as classifying coloured objects Q4-(P2). The workload in each hand for co-ordination requires a certain degree of competence for manipulation and both are complementary. Co-ordination therefore reduces muscle fatigue and enlarges the reachable area through a small movement of exchanging hands. In a serious 2D or 3D game environment, these hand functions would be frequent actions operated along the XYZ axes in the game space. Regardless of the stage of dementia, residents in the nursing home could complete most of the hand functions (Fig. 6), but relatively fewer residents could complete the co-ordination function as it requires sufficient tolerance for completion and the duration of the task is generally assigned based on the patient’s tolerance. Therefore, co-ordination could be a hand function that cognitive-impaired persons have difficulty with. Other parts of the upper limb functions are also relevant to the framework since they enable expansion of the training area and physical movements of an MCI player within a serious game, notably: wrists for twisting, pressing and tapping; elbows for pulling, which involves flexion and extension movements that enable objects in front to be raised to a particular point; and shoulders for translating the whole upper limbs to perform activities. These enhance controllability (see Section 5.2) through intact upper limb movement and can further test the player’s ability to manipulate objects. For example, wrist capability can help determine if the finger muscle mechanism is normal, while shoulders support different subtle motions of the player during tasks even where those tasks could be carried out without them.

Lower limb capabilities comprise balancing while sitting or standing and were assessed as poor, fair or good within the nursing home residents (Figs. 7, 8 and 9). Typically, patients are able to balance better when sitting rather than standing, therefore, it is likely that an MCI player would undertake game-based rehabilitation while sitting. However, the profile serves to record both capabilities so that a serious game may be designed accordingly.

Based on the OTs’ experiences, it is evident that some MCI patients may have visual or hearing impairments Q1-(P2, P4), which may directly affect their sensory capability to process experiences. Recording this within the profile enables a serious game to adapt the therapeutic content accordingly, e.g. by enlarging images and amplifying sounds so that patients are better able to process the information. These are common methods to capture and sustain their attention Q1-(P1, P4), Q2-(P4), Q5-(P2, P3), interest and active participation.

Current nursing homes and hospitals prefer to also include a wide range of functional tasks (all basic functional tasks) when examining a patient’s physical ability. The tasks should reflect day-to-day activities since patients have already acquired great proficiency in basic physical movements such as dressing and cooking gestures. As reported by the OTs, patients who have no medical condition such as brain injuries generally have sufficient mobility competencies to perform all basic physical movements Q4-(P1-P4) including upper and lower limbs movement, i.e. sitting and standing balancing. Including the MCI player’s level of capability for such tasks within the profile of the framework will ensure that the game is able to appropriately challenge the player.

5.1.3 Player cognitive capabilities

In the nursing home, residents were assigned a range of functional tasks or personalised training, which included cognitive functional tasks targeted at the five main cognitive functions: attention, working memory, use of language, visuospatial skills, and executive functions. Therefore, the player profile reflects their capability in each of these functions. The interviews revealed that attention is required at the beginning of cognitive rehabilitation, and it is one of the more challenging cognitive domains Q5-(P1, P4) because sustaining patients’ attention during the entire rehabilitation or therapy is difficult but crucial. In addition, prior to attention, the cognitive process requires sensory registration as a basis for processing the quality and quantity of visualisation, as seen in the example of using colours to stimulate the patients Q5-(P2). The attention level of residents in the nursing home (Fig. 3) varied from the range “unable to concentrate” to “easily distracted” to the range “easily distracted” to “attentive,” with most residents assessed as being “easily distracted.” Those diagnosed with MCI were in the range “easily distracted” to “attentive,” indicating that attention support will also be required as a motivational element to maintain a good cognitive condition for the MCI player to concentrate on the tasks, and maintain their interest to prevent them giving up. This will need to vary according to the player’s level of attention. Working memory is the most used cognitive function that allows for the temporary storage of information. Most of the MCI residents in the nursing home were assessed as being capable of processing working memory to complete tasks by recalling information, and were consequently assigned to individual memory training so as to train these capabilities. Use of language is normally assessed prior to other functions to ensure that the patient is able to verbally communicate normally and fully articulate themselves. Together with attention, these cognitive capabilities are considered the fundamental abilities to support MCI through the functional tasks. Visuospatial skills and executive functions for MCI were typically examined through ADLs or IADLs in the nursing home. During the free discussion within the interviews, the OTs believed that those with MCI could typically manage all of the main cognitive functions and that cognitive training was shown to improve memory functions. Consequently, they argued that long-term memory support could improve an MCI player’s working memory impairment.

5.2 Core gaming elements

All core gaming elements within the framework were predominantly derived from the literature as per Tables 1, 2 and 3, in terms of both gamefulness and playfulness. Within gamefulness, player’s perspectives, tutorial & guidance, player’s progress & rewards, and metaphorical graphics may be used to assist the MCI player in understanding their status within and the context of the serious game. These elements are responsible for generating the vision, steps, and personal progress of the player which conforms to their actions within the game. Flow is important for keeping the player motivated so that they can perform tasks with enjoyment, focus and understanding in order to be immersed in the game. Flow can also help to reinforce the concentration of MCI patients and cultivate long-term participation. Pointsification is used to facilitate extrinsic motivation, for example, through redeemable points or badges after the MCI player has completed a level in order to lead them to the next level. In order to ensure that such rewards appeal to the MCI player they should be closely-related to their lifestyle, such as badges that relate to hypothetical roles they may adopt within a nursing home among their peers, e.g. “card game watchdog badge.” HUD design serves to visualise and contextualise these gameful elements appropriately to the player in the form of appropriate visual cues that the MCI player is able to quickly understand in order to focus on the challenge.

Some elements exist as subsets of both gamefulness and playfulness. Personalisation enables the serious game to be adapted according to the MCI player profile. Feedback provides guiding information to the player which may serve to motivate, instruct, or similar. Narratives may be used to provide context to the therapeutic elements so as to further aid the player in relating to real-life situations or reminiscing. The 2D/3D environment and avatar-based elements facilitate the construction of the game space and how the player exists within that game space. The individual care plans from the nursing home indicated that residents were commonly assessed as having a reduction of physical controllability, such as mobility issues and stiffness of limbs. Furthermore, in the interviews, the OTs indicated that most patients have sufficient competence to manage and understand new information including digital platforms for rehabilitation/therapy Q9-(P1), but it is recommended that the content is relevant to the therapeutic routines. Furthermore, the complex functionality of digital games should not be an additional burden or obstacle that prevents patients from understanding the therapeutic context Q9-(P1). For example, a mouse and keyboard are unlikely to benefit those with no computer experience Q9-(P1, P4). Thus, it is important that the range of interactions and physical movements within the therapeutic game environment are accessible and compatible with the MCI player’s abilities. Consequently, controllability not only reflects the interaction modality but also adjustment of game object sensitivity so that less or more movement is required as necessary, working space flexibility so that the game environment can facilitate working space sufficient to accommodate the limb power of the MCI player, and freedom of movement so that the MCI player is able to navigate in-game scenes with as much freedom as their limbs are able to accommodate, e.g. via full upper body gestures, which helps bring a sense of control and positive satisfaction.

The framework contains four elements that are expressly organised within playfulness: behavioural flexibility, affection, contextual awareness and self-representation. These elements are highly focused on stimulating the MCI player’s preference for the activity. Behavioural flexibility enables control of the exploration of the gaming scenario, while contextual awareness may be used to help elicit greater understanding of the game space and possible player behaviour within it in conjunction with other game feedback. Self-representation is the form of performance in an activity in which the player’s attitude is taken into account and can be utilised with elements of affection and the MCI player profile, so that the serious game can offer a competent and meaningful role for the player to engage with, which may be manifested through avatar-based playful scenarios that the MCI player would intrinsically be interested in.

5.3 Therapeutic elements

Therapeutic tasks and therapeutic scenarios are complementary in serious games and therefore are grouped together within the framework. Therapeutic tasks are provided to the player in accordance with therapeutic scenarios and both must be delivered within the game environment according to the MCI player’s capabilities. In the interviews, the OTs reported that they mostly assign a variety of tasks that comprise cognitive and physical rehabilitation to patients in accordance with their medical history, pre- and post-assessment results and intact profiles Q3-(P4). Based on the therapeutic tasks, the patients will then be assigned various scenarios in accordance with their medical conditions.

5.3.1 Therapeutic tasks

Numerous types of cognitive and physical rehabilitation were compiled from the related research literature as reported in Section 2 and were presented in Tables 1, 2 and 3. In the framework they are classified into physical functional tasks, main cognitive tasks, and other physical tasks. Both physical functional tasks and main cognitive tasks were adopted within the nursing home where ADL is a part of the resident’s assessments. Typical physical tasks include having meals, transferring, and grooming, which highly involve the upper limbs (both hands). Upper limb tasks are also commonly employed for therapeutic use in the research literature reviewed in Section 2, such as IADL-themed tasks, and all upper and lower limb tasks which were used in the reported games were manageable for MCI patients. Similarly, some lower limb tasks are also essential to daily life, such as sitting and standing balancing, and are used within the nursing home. Residents who often fail to maintain an appropriate sitting and standing position will face difficulty in the completion of therapeutic tasks. All upper and lower limb tasks used in the nursing home are suitable for MCI patients. In the interviews, the OTs suggested that requesting patients to perform table tasks (working with upper limbs) with a good sitting position can reduce their physical limitation Q9-(P1,P4), i.e. lower limb ability, as well as enhance their attention and tolerance levels. Some therapeutic tasks utilised other parts of the body and these other physical tasks (e.g. fall prevention and oral muscles tasks) were not necessarily targeted at MCI patients though may be relevant in certain circumstances.

It was noted that cognitive and physical rehabilitation tasks should be examined separately Q9-(P1) and therefore main cognitive tasks serve to complement physical functional tasks. Most of the cases with memory impairment in the nursing home were reported as undertaking working memory training. In the interviews, the OTs reported that they would assign all cognitive training to their patients, with working memory training given to those with memory impairment in accordance with their pre- and post- assessment history records. Working memory is the specific cognitive function used to identify the MCI group. However, prospective memory and other memory training may also be targeted, e.g. where the player is required to keep multiple thoughts in mind and switch between them while undertaking multiple therapeutic tasks within a given IADL scenario.

5.3.2 Therapeutic scenarios

The therapeutic scenario is created for the player to undertake the challenge of the tasks by mastering the orientation of the situation. For example, a supermarket simulation [57] presents a situation enabling a player to buy things using their perception. It allows the player to become aware of the context and therefore the scenario mostly provides visual cues to the player to integrate into their cognitive map. Many different scenarios can be used in a therapeutic way. In order to ensure that the therapeutic scenarios are appropriate for MCI players, we incorporate those therapeutic scenarios within the framework which are commonly assigned to MCI patients, e.g. in nursing homes and other medical centres.

In the nursing home we investigated, the therapeutic scenarios mostly utilised to treat residents with MCI were ADLs, IADLs, reminiscence and reality-oriented scenarios as these were deemed to have the highest positive effect on the target group. Generally, MCI residents in the nursing home achieved a standard level in ADLs and thus this scenario may be used as a basis for integrating more challenging functional tasks into a serious game. IADLs in the nursing home provided more complex tasks for those in mild dementia cases but may pose difficulties for those with later stage dementia. However, related research in Section 2 demonstrated that close-to-reality approaches, i.e. IADLs and Reality Orientation, have a positive effect on those with cognitive impairment. To facilitate a serious game for MCI players, both approaches can be employed since they require more complex physical and cognitive management skills which the MCI group is able to undertake. The individual care plans from the nursing home revealed that residents’ cognitive abilities could deteriorate further due to old age and extended hospitalisation, and long-term confinement to the bed and lack of social stimulation can negatively affect their cognitive state. Therapeutic scenarios with an environment assessment context and reality-oriented activities can enhance the player’s environmental awareness and acknowledgement of reality-oriented information, such as time, location, date and weather. Such a scenario would aid the player in deriving their player status from the gaming environment through visual gaming elements, such as warning signs, weather/clock icons. In order to improve environmental awareness, colours can act as indicators, e.g. red for danger. This supports the player in accomplishing and understanding the assigned tasks by providing memory aids and orientation information. In the reminiscence scenario commonly used in the nursing home, cultural elements were embedded into the tasks with the aim of encouraging residents to recall the past. The fact that the scenario setting is closely related to the resident’s personal experience and background can help to induce the patient to recall long-term memory.

The interviewed OTs reported that numerous types of memory training are frequently integrated into a comprehensive rehabilitation design, the majority of which are targeted at improving working memory (short-term memory), which primarily declines with the increase in age. Instead of merely focusing on memory during training, integrating memory training elements with other training elements like reality orientation, ADLs/IADLs or a simulated setting (e.g. reminiscence) can enhance the cognitive training since the other training elements already involve working memory Q8-(P1-P3). Additionally, long-term memory can further support working memory functioning by enabling patients to relate to the context of the setting. For example, in a shopping simulation, patients use long-term memory to recall their role as a consumer in a supermarket, while they use short-term memory to remember a list of items to buy Q8-(P4).

5.3.3 Motivational elements

Motivational elements within the framework are intended to enhance player attitude towards the serious tasks and to encourage the player to be more explorative. As well as general motivational elements, various playful activity, gameful cognitive rehabilitation and gameful physical rehabilitation elements were identified from the literature and identified in Tables 1, 2 and 3.

The OTs’ experience is that the duration of a therapeutic session is typically 30–45 min Q3-(P4). There are an optional 15 min of warm-up and 10–15 min for each rehabilitation task, with no break in between Q3-(P1). While some patients are willing to participate in a session of up to 60 min, those who feel unmotivated might only last for 15 min Q3-(P2). The variation in duration therefore should be tailored accordingly within the serious game. In addition, if patients have a sudden medical change during the cognitive rehabilitation or therapy, it will be uncertain whether they would be able to continue with the session Q5-(P3), which the serious game will need to accommodate. The learning approach may help to counter problems of being easily distracted and losing focus, which may result in tasks being unfinished. Providing an errorless learning environment, which enables the player to focus on continuing to perform in the task in the absence of displaying errors [33], is perceived as being both an attention support and motivational tool, which hides feedback on errors during the session to enable patients to focus on continuing Q2-(P4). Similarly, the OTs reported that attention support is essential for patients to be initiated into a rehabilitation and effective attention support can help patients to be focused on making the right decision without straying from the rehabilitation/therapy. Clear instructions which help patients visualise the training context can facilitate the explanation of the training goals and increase attention level Q5-(P2,P3). Furthermore, attention support can help to arouse and scaffold the patient’s interest and attention during the rehabilitation/therapy if the content is closely related to their daily lives and personal preferences Q1-(P4). However, patients may feel uninterested and unmotivated if too much effort into learning new knowledge or techniques is required Q1-(P1). Attention skills can support the visual and spatial perception of patients to aid in anticipating the next action. Consequently, working memory ability is required to temporarily store information for executing the actions once the patients can recognise the objects and receive the information through their senses. Therefore, sustaining patients’ attention is a precursor to using their working memory ability. Supervision and verbal encouragement, typically from one trainer, can benefit the patients in social and mental ways Q2-(P1). The biggest impact of technological support in rehabilitation is the increased precision of patients’ quantitative analysis results Q7-(P1,P3). However, technology cannot provide better observation and analysis than an OT in more subtle areas, such as the behaviours and emotional quality of patients Q7-(P1-P3). Therefore, the serious game should assist the OT during rehabilitation but not fully replace them Q7-(P3). The technology can also reduce potential risks and enable diversity Q7-(P1) but may greatly reduce the social interactions between the trainer and patient Q7-(P2). In some cases, home-based patients might not have a chance to participate in certain technologically-based rehabilitation since they would need to travel to the nursing home or hospital Q7-(P4). In general, to achieve the best training effect, the serious game should incorporate a role for trainer supervision and verbal encouragement. Even when patients have much interest and know-how regarding computer-based therapy, a well-maintained relationship between trainer and patient can further promote cooperation leading to a desirable training effect Q1-(P4). In respect of rehabilitation equipment, providing patients with tangible tools to touch or control objects with sensory feedback can improve their motivation towards the tasks Q1-(P3), Q2-(P2) and therefore should be accommodated within the framework.

Within the playful activity motivators, discoverability and autonomy: freedom of choice can potentially support patients in performing tasks more effectively compared to traditional training platforms due to the diversity of the digital platform Q6-(P2-P4) and the geographical constraints of traditional training Q6-(P3). With the former, various types of rehabilitation/therapy are highly compatible with patients’ preferences and capabilities, and patients are less likely to find tasks tiresome compared to traditional training platforms. The serious game may produce notifications, hints or trials to patients that are not available via traditional approaches. With the latter, geographical constraints may be overcome by avoiding the need to utilise real space settings, for instance, in reminiscence therapy. When relatedness elements are also employed, the serious game may support collaborative or co-operative play with other local or remote patients.

Core to all motivated gameful and playful activity is the need to foster competence: skills, challenges and immersion: player’s experience so that the player is psychologically immersed within the gameplay and sufficiently encouraged to proceed with skills and challenges that are ‘just right’ for them. The gameful cognitive rehabilitation motivators of simplicity of game objects and realism of graphic model enable the player to feel suitably engaged according to the requirements of the task and scenario and their own abilities. For physical rehabilitation, the use of real-time game objects is important so that the player’s physical movements are synchronised with the behaviour of their character or avatar within the game world.

For both gameful cognitive and physical rehabilitation motivators, sorting and grading the patients according to their cognitive levels is a way to motivate them to achieve a clear goal, as well as improve a sense of competence Q2-(P1-P4). These patients’ capabilities will be fully recorded in the MCI profile according to their medical history and thus the level of tasks may be adjusted accordingly Q3-(P4) and, in the long-term, therapeutic content may also be updated. The use of achievements as motivation may be used in conjunction with other gaming elements, such as pointsification which would help to instantiate and visualise those achievements to the player. Distractors are key for ensuring that therapeutic tasks are suitably challenging. Together with levels may serve to help increase their complexity for the player, e.g. by hindering or preventing them from undertaking a task in a straightforward manner.

In the next section, we use the framework to develop an immersive serious game for MCI.

6 A-go! An immersive serious game using MCI-GaTE

To demonstrate the use of the above framework, we developed A-go!, an immersive serious game targeted at an MCI player and their OT. The game is only intended to be used in conjunction with supervision from an OT since they are best placed to assess the performance of an MCI player and due to the vulnerable nature of the players. In this way, the game is intended to be used in a similar setting as existing rehabilitative tools but to improve patient motivation and engagement over such tools. We consider immersion as “the subjective feeling of being enveloped by the games’ stimuli and experiences” [53]. Therefore, A-go! exploits gesture recognition and 3D game objects from a first-person perspective to facilitate immersion without requiring any physical HMD (head mounted device), which would prove impractical for the targeted elderly players. The OT is responsible for supervising the player, including demonstrating the tasks, and verbally supporting them throughout the game. This support is enhanced by a registered medical doctor, who serves as an external participant. The doctor makes the MCI diagnosis and passes it to the OT for review of background information and player capabilities. The game works in conjunction with a standardised pre-assessment screening (that works complementarily with a post-assessment), to assess the player’s cognitive and physical qualities which are used to populate the player profile.

6.1 Use of MCI-GaTE for A-go!

Even though the full MCI player profile is incorporated into A-go!, we noted that the nursing home residents had a poor condition in standing balancing. Moreover, the OTs revealed that upper and lower limb tasks tend to be assigned separately to the patients due to cognitive load concerns. Consequently, the lower limb tasks incorporated into A-go! are sitting balancing only so as to optimise the patients’ cognitive capabilities under light physical interaction conditions. In addition, other physical tasks are not deployed as only MCI patients who are in late-stage dementia may manifest signs of other physical impairments affecting mobility and gait. Therapeutic scenarios are presented as ADL, IADL, reminiscence and reality orientation activities, to reflect those used in conventional rehabilitative therapy.

From the core gaming elements, a first-person player’s perspective has been adopted throughout to enable the MCI player to focus on their tasks with the designated hand gestures, thus A-go! is not avatar-based. Tutorial and guidance and HUD design are primarily manifest in the form of a demonstration phase, with a minimal HUD design subsequently to provide some contextual awareness and report player’s progress and feedback in a manner that reduces cognitive load and supports the adopted errorless learning approach. Thus, pointsification, player’s rewards and achievements (from motivational elements) are not employed. Metaphorical graphics serve to assist the MCI player in understanding their status. Overall, flow helps to sustain the MCI player’s motivation and enjoyment, supported by implicit use of narratives within the context of individual therapeutic scenarios. In terms of 2D/3D environment, A-go! is predominantly a 3D game space with 2D virtual hands in order to help the MCI player more readily distinguish their movements. Together with personalisation and controllability, these elements are complementary to each other, providing behavioural flexibility to enable the MCI player to manipulate the game objects via their own hand gestures with responsive feedback according to their personal capabilities. Affection and self-representation are not specifically apparent, but the use of reminiscence-based therapeutic scenarios is likely to engender them somewhat.

From the motivational elements, since supervision and verbal encouragement are provided by the OT, A-go! adopts duration guidance rather than time limits (15–20 min for demonstration and 15–60 min for therapy) and uses essential on-screen prompting and feedback. Discoverability and autonomy: freedom of choice echo behavioural flexibility to allow the MCI player to intuitively explore the game scene. Distractors are reflected in the game’s levels and player’s competence: skills, challenge. However, the distractors are simple since MCI players may be more sensitive to them due to their generally lower attention level. Similarly, simplicity of game objects is highly utilised throughout the game to enable players to focus, albeit in conjunction with realism of graphical model so that objects may be recognisable by the MCI player as those from daily life, depending on their cognitive load and attention level. Due to the standing balancing physical tasks not being adopted at this stage, tangible tools are not utilised. However, real-time game objects ensure that the player’s hands are synchronised with those on-screen and help to facilitate the sense of immersion: player’s experience. While the involvement of the OT in providing support at all times may provide some notion of relatedness: cooperation, social collaboration, A-go! is not a co-operative or collaborative serious game per se as this would require assessment comparisons to ensure players of similar capabilities.

6.2 Architecture and gameplay of A-go!

A-go! runs in Unity 2019 in a macOS desktop environment to provide appropriate controllability given the common profile of MCI patients. The 2D and 3D assets for the game were created in Autodesk Maya 2019 and Adobe Illustrator CC. Interaction is via a Leap Motion controller using the V2 SDK and gesture recognition to facilitate first-person perspective immersive interaction and experience without requiring a wearable physical device. To perform the player’s gestures, the virtual hands are navigated on the desktop display through the infrared image recognition of Leap. The MCI player and OT are required to place their hands upon the device and apply the designated hand gestures freely to interact with the 3D game objects in real-time. The increase in a sense of controllability encourages the player’s hands to freely interact with the 3D game objects within the 3D space. Provided with 3DVE and realistic adherence with 2D UI motivational elements, it creates an intuitive series of therapeutic tasks that allow the player to choose an action among eight designated hand gestures with the responsive core gaming elements. The hand gestures imitate recognisable daily-life hand functioning as used in the nursing home. It reduces the duration of task-switching and speeds up the learning process, as well as providing high satisfaction of control. Hints are used to direct the player when necessary, e.g. dashed red lines around an object serve as memory support. The architecture of A-go! is illustrated in Fig. 11 and is now discussed.

The game environment commences with a demonstration phase to help the MCI player, supported by the OT, gain familiarity with the environment before undertaking the tasks. This phase typically lasts 15–20 min, according to patient capabilities and needs so as to maximise their participation in the subsequent therapeutic scenarios. Thus, the MCI player profile enables the demonstration and the game-based training to be pitched at the correct level for the player’s diagnosis (to achieve the optimal training effect), to ensure that the physical and cognitive capabilities are tested by completing the substantial hand gestures during the demonstration as presented in Fig. 12. The purpose and benefits of the game are explained, with the OT verbally supporting and encouraging the player. The demonstration tasks involve the player placing 3D game objects on a target by performing the necessary gesture. The OT will demonstrate the common daily hand gestures to the player (opposition, fist, grasp and release, pinching, coordination, pressing/tapping, pulling, translating), where the opposition gesture comprises the confluence of fine motor skills incorporating the other gestures. Thus, once the player successfully completes the opposition demonstration, they will manage to complete the remaining gestures. Therefore, it serves as a quick form of screening to ensure that the player has sufficiently intact upper limb (hands) capability. The player is assigned to do the tasks with upper limbs when sitting still, thus the lower limbs are not involved during the training. As previously discussed, both upper and lower limb training usually operate separately to allow the player to focus on cognitive functioning with light physical interaction.

Fig. 11
figure 11

The A-go! immersive serious game

Full size image
Fig. 12
figure 12

Demonstration phase

Full size image

Four themes based on therapeutic scenarios are then employed in the game to mimic real-life activities using various gaming elements and incorporating suitable gestures: (i) Sandwich Making as in Fig. 13, which is an IADL, (ii) Toileting as in Fig. 14, which is an ADL, (iii) Reality Orientation Board as in Fig. 15, a reality orientation scenario, and (iv) Mahjong as in Fig. 16, a reminiscence scenario. These serve as relatable and familiar scenarios where the MCI player can train. This rehabilitative training typically lasts for 15–60 min according to the player’s capabilities and as overseen by the OT, and reflects the typical durations specified in the interviews Q3-(P4).

Fig. 13
figure 13

Sandwich making game scene

Full size image
Fig. 14
figure 14

Toileting game scene

Full size image

The game level maps to the MCI player’s physical and cognitive capabilities from the profile, which are adjusted during post-assessment with the OT which occurs after each game session, so as to determine the level for the MCI player by altering the complexity of the tasks. Beginner level is aimed at a novice who is inexperienced in the serious game and requires clear and detailed explanations from the OT due to their lack of background. Step-by-step guidance ensures that they are able to apply the hand gestures and concentrate on a single game without excessively being concerned about the game utility (which would increase the cognitive demand). Intermediate is aimed at more experienced players who understand the gaming context and have previously managed to complete game tasks correctly at the beginner level. Thus, they are qualified to consecutively accept the challenge of taking on two games. The advanced level is for players who can complete the intermediate level and have been making progress from the previous game results, suggesting that they are able to achieve positive progress in serious tasks and undertake three games in a row. Figure 17 reflects the three games that would be utilised in these levels for the Sandwich Making theme. For the Toileting theme, the games are Game 1-Toilet flush, Game 2-Hand wash and Game 3-Paper towel; for Reality Orientation Board, the games are Game 1-Change weather, Game 2-Tune the clock and change weather and Game 3-Tune the clock, change weather (with more options), and for Mahjong, the games are Game 1-Missing one tile, Game 2-Missing two tiles and Game 3-Missing three tiles.

Fig. 15
figure 15

Reality Orientation Board game scene

Full size image
Fig. 16
figure 16

Mahjong game scene

Full size image
Fig. 17
figure 17

Game levels

Full size image

To enable the physical (hand gestures) and cognitive tasks during the training, each game theme requires the player to recognise the need to perform and carry out two to three hand gestures as they would need to do if controlling the objects within the task in real life. In Sandwich Making (Fig. 18), pinching and coordination gestures are adopted for meal preparation training. This commences with operating the press handle of a toaster in Game1-Food where the player is required to pinch down the press handle to toast the slices of bread and pinch up when the bread is sufficiently toasted. Pinches up is then extended in Game2-Drink to involve both hands to simulate an orange being squeezed and poured into a blue mug. Game 3-Dishware requires the player to select the correct plate through grasp and release gestures. The Toileting theme (Fig. 19) requires the player to press and hold the flush plate momentarily in Game 1-Toilet flush, and to memorise the amount of hand wash and paper towels in Game 2-Hand wash and Game 3- Paper towel respectively, with pressing/tapping and pulling gestures to perform the actions. Two hand wash boxes are placed at different positions as distractors to train the user’s attention. The Reality Orientation Board theme’s tasks (Fig. 20) challenge whether the player is able to understand real-time surroundings. In Game 1-Change weather, translating and pinching gestures are required by the player to ‘change the weather’. This is combined with tuning the clock in Game 2-Tune the clock and change weather. Game 3-Tune the clock, change weather (with more options) provides all elements from Games 1 and 2 with a greater number of weathers to increase the difficulty. Mahjong (Fig. 21) requires the player to use the grasp and release gesture. A series of Mahjong tiles are displayed with 1–3 tiles missing according to the level of the game, and the player is required to grasp the correct tile from the centre of the table to complete the task. Reminiscence is introduced to support the cognitive functioning by drawing the MCI player’s attention. These tasks focus predominantly on memory to support the impairment of memory in an MCI player, especially their working memory (short-term memory), while sustaining their other cognitive capabilities, in line with the research discussed previously.

Fig. 18
figure 18

Sandwich making - pinching and coordination gestures

Full size image
Fig. 19
figure 19

Toileting - pressing/tapping and pulling gestures

Full size image
Fig. 20
figure 20

Reality Orientation Board - translating and pinching gestures

Full size image
Fig. 21
figure 21

Mahjong - grasp and release gesture

Full size image

The game scenes visualise the gaming context and current status to the player, with additional text for the OT who will additionally support and motivate by providing comprehensive instructions throughout. To distinguish the player’s movements from the 3D game objects, the virtual hands with orange gestures are in 2D.

As with conventional training, the game environment is errorless to enable the player to focus on the tasks without interruption until the end of the game. Thus, while some prompting may be used as attention support to guide the player to take action, error feedback is not shown at that time. Instead, errors and scores are displayed at the end of the game session as shown in Fig. 22. These results, together with those gathered in game, are evaluated by the OT as part of a post-assessment, shown in Fig. 23, which is used to inform adjustments to the player profile for subsequent gaming sessions which will consequently adapt the game. For example, this will involve determining the player’s flexibility in using hand gestures, their understanding of the given therapeutic scenarios, their working memory functioning in remembering a list of items (e.g. sandwich ingredients), and so on. The adaptations involve difficulty adjustment via upgrading or downgrading of game levels, or repetition of training tasks. To enable the player to progress to a higher level, their scores must reach 80 % or higher of the total score, while the player is assigned to repeat the same level when achieving 50–79 %, and downgraded if below 50 %. The serious game would be played once or twice per week to maintain the training progress, with more extensive post-assessment taking place monthly or bi-monthly to validate the player’s training progress. Improvements in post-assessment score would indicate the positive impact of the serious game on the MCI player.

Fig. 22
figure 22

Scores and errors

Full size image
Fig. 23
figure 23

OT dashboard

Full size image

6.3 Evaluation of A-go!

The game is designed to be used in a similar setting as existing rehabilitative tools, that is, in conjunction with an assigned OT, so that MCI patients are best supported. Therefore, OTs who deal with MCI patients daily are best placed to assess A-go! as one of the tools they may use in their work. We therefore returned to the OTs interviewed in Section 4 who between them have considerable experience observing MCI patients over prolonged periods, assessing their physical and cognitive states based on screening tools and protocols, and designing appropriate interventions for them. Thus, they are highly able to provide rigorous validation of the proposed serious game. Additionally, MCI patients are a vulnerable group who are likely to be unduly distressed by such an evaluation and their cognitive impairments, e.g. forgetfulness, would lead to inaccuracies and omissions in the results.

The OTs were invited to test A-go! thoroughly for an hour, from demonstration through to the scores and errors stage, during which the concurrent think-aloud (CTA) technique [52] was used. Identified issues and comments were recorded and raised during subsequent semi-structured interview sessions, which were scheduled on two separate days totalling 2–3 h per OT. The following open-ended questions were used to validate A-go!’s effectiveness for improving MCI patient rehabilitation:

  1. Q1.

    Which stage(s) of the proposed serious game do you believe that MCI players would need further explanation from you, e.g. demonstration, utility of the game, new environment, etc.?

  2. Q2.

    What signs immediately tell you that it may be necessary to support MCI players during the game? Do you think players can understand the UI designs easily?

  3. Q3.

    Should any elements/features in the game be removed or added in order to improve the training (cognitively and physically)?

  4. Q4.

    To what extent do you think the proposed serious game can improve MCI players’ motivation and engagement through the gameful elements? Why?

  5. Q5.

    What advantages and disadvantages does the proposed serious game have compared to the traditional training method?

  6. Q6.

    In terms of a-MCI, do you think the game can potentially sustain MCI players’ (episodic) memory capacity after a certain period of training? To what extent and why or why not?

Screenshots of the game interfaces were provided for reference and to prompt further discussion. The interviews were carried out in Cantonese and the responses translated and transcribed into English verbatim.

The interviews revealed that the first challenge of the entire intervention journey is the initiation of the MCI players into the game. The majority of OTs acknowledged that a very clear prelude to the game scene can help MCI players understand their participation goals and diminish their hesitation to participate, especially new players Q1-(P1-3). Demonstration is also important for introducing game functionality Q1-(P1-3), Q2-(P2) and hand gestures to the player Q1-(P2), which allows the OT to demarcate mistakes arising due to the player not understanding the game functionality and those due to the player’s declining cognitive and physical capabilities Q1-(P3). To enhance motivation and engagement, it is also necessary to create a “training atmosphere” by emphasising the training purposes to the player (e.g. supporting cognitive functions Q3-(P4)) and the context of the game Q1-(P1). It is also worth drawing attention to the advantages of the game, such as the beneficial outcomes on particular muscles and memory ability, so as to further raise the player’s awareness of the therapeutic goals Q1-(P4). Such aspects may be carried out before or during the demonstration phase by the OT.

The hand gestures used in A-go! were noted as being suitable and manageable for MCI patients, but OTs would still need to assess the player throughout the game, particularly on their use of the gestures Q1-(P2). Additionally, the player’s training history, physical constraints (visual and auditory problems) and education level, as represented by the MCI player profile in MCI-GaTE, were confirmed as being suitable for providing effective support and anticipation of the player’s needs Q1-(P4). However, it was suggested that the prominence of hand interaction in the game could be further emphasised in a future version by appropriately stimulating the player’s physical senses Q3-(P2), e.g. through haptic feedback, which may help overcome some physical constraints for the MCI player. This would relate to the tangible tools element of the proposed framework.

All OTs confirmed that the selected gaming elements in A-go! can potentially overcome significant clinical difficulties and the constraints of traditional rehabilitation tools, effectively supporting the MCI player and improving their motivation and engagement as they undergo the training Q4-(P1-P4). In this way, MCI-GaTE and A-go! can potentially outperform the traditional methods. The game’s novelty can unleash a player’s curiosity and motivation towards the game which appears to be sustained through responsive gaming features and in-game support by the OT. The following were noted as being of particular relevance: the variety of game themes Q5-(P1), the personalised features Q5-(P1), the high flexibility in operating the training for home-based patients as well as those in clinical settings Q5-(P2), and the simple setup which reduces preparation time compared to traditional methods Q5-(P1,P3, P4). However, there may be some players who have a specific preference for the traditional methods due to familiarity and a preference to avoid the stress of new technology Q5-(P1-P2). The OTs also saw distinct advantage in the automated feedback which can reduce the manual overhead Q5-(P2) and enable the OT to focus purely on player support during the therapeutic session. Potentially, the game may maintain the player’s memory capacity Q6-(P2) and achieve positive results Q6-(P4) after a suggested period of training of twice a week Q3-(P3) over a period of 4–6 weeks Q6-(P4). Some also reported that the skills the player acquired from A-go! may be transferable to real life Q5-(P3) particularly with respect to the player’s upper limb competence Q5-(P2, P4). Additional use of haptic feedback would further support this Q5-(P4), Q3-(P2). However, it was suggested that A-go! be further assessed against standardised screening tools Q6-(P1-P3) and that the player’s results of pre- and post-assessments should be compliant with the standardised scoresheet Q6-(P2).

Overall, the OTs concluded four factors that may be indicative of the player’s motivation and engagement. Firstly, the interaction between the player and the interface during the game is crucial Q1-(P4). The freedom of hand movements within the virtual environment of Ago! motivates the player to acquire an exploratory attitude towards the game Q4-(P1), while the incremental adaptation of game content and level each time the game is played is likely to further sustain player interest Q4-(P2). Secondly, the cultural relevance within A-go! is an important motivator Q4-(P4), e.g. the Mahjong game scene is close to the players’ daily routines whereas the Sandwich Making game scene is comparatively less familiar to them Q4-(P3-P4). However, Sandwich Making creates an opportunity for the player to cook safely (operate the toaster) Q4-(P3) and eliminating the risk of injury may also serve as motivation. Thirdly, players are more likely to accept the game if they have similar prior experience Q4-(P4) with digital games, suggesting that a demonstration phase or ‘trial run’ of the game should be carried out in the days before the training proper. Fourthly, although A-go! is personalised for an individual MCI player to undertake the training, some players may prefer not to play alone Q1-(P1), thus a multiplayer option would serve as further motivation Q4-(P4). This would necessitate involving those with a similar game level to generate a collaborative gaming environment via the relatedness: cooperation, social collaboration element of the framework. The presence of the OT also serves as an important external motivator to help the player relish the training Q2-(P1).

A crucial purpose of the inbuilt core gaming elements is to convey the therapeutic content to the player. The OTs agreed that those within A-go! are understandable to an MCI player, but several refinements were proposed to further facilitate the player’s learning speed and adaptability. It was suggested that allowing the player to engage for longer may enable them to better learn the interface elements and context by themselves Q1-(P3), Q2-(P2). This is already accommodated in A-go! since no timer is embedded. More explicit in-game instructions may also further support the player’s understanding in line with their individual attention span Q1-(P4). For instance, the utility of the toaster which an elderly player may not be familiar with (as they were not prevalent in Hong Kong for domestic use) could be demonstrated Q4-(P1). The accessibility of the game can also be elevated through further realism Q2-(P1), such as realistic hands, game objects Q2-(P3), and a 3D simulated kitchen setting Q3-(P1). Moreover, the game hint (dashed line) on the toaster could be in sharper contrast Q2-(P1), Q3-(P1) to better alert the player to the feedback, and a motivating indication of completion, such as “Good job!”, could be displayed Q3-(P4). In terms of the level adjustment, it was also advised that the game offer a few gestures at a time since the player is less likely to recall all the gestures learned during the demonstration Q2-(P1).

The OTs also pointed out several exceptional situations which may occur during the game: inappropriate operation of the given equipment Q2-(P1), pausing for a considerable amount of time in the middle of a task or even opting out of the game Q2-(P1-3), and constantly asking questions Q2-(P2, P4), the latter of which would manifest with significant memory or attention issues Q2-(P4). It was not felt that these could be accounted for by the game and given their rare and anomalous nature would best be managed by the OT as part of their in-game supervision and support.

Overall, the evaluation revealed that A-go! could be more effective in supporting MCI patients than traditional training tools, but that further assessment through standardised screening tools should be undertaken. The addition of further features, such as haptics and multiplayer options, would facilitate additional effectiveness and engagement. In the next section, we undertake a comparative assessment of MCI-GaTE against relevant serious game frameworks.

7 Comparison of MCI-GaTE against existing rehabilitative serious game frameworks

There are many challenges to developing a comprehensive serious game framework for rehabilitation due to currently inconclusive research regarding target player, gaming and motivational features pertaining to specific rehabilitative contexts. To further demonstrate the significance and efficacy of MCI-GaTE and A-go!, we consider the framework against existing physical and cognitive rehabilitation frameworks for serious games. Table 5 compares MCI-GaTE with 11 contemporary serious game frameworks for physical and cognitive rehabilitation according to the degree with which they incorporate player profiles and therapeutic, core gaming, and motivational elements.

Table 5 Comparison of frameworks with MCI-GaTE
Full size table

As can be seen, the majority of proposed serious game frameworks target physical rehabilitation only [1, 6, 8, 18, 25, 27, 35, 46]. Several of these [1, 8, 25, 27, 35] construct a player profile by investigating player’s characteristics pertaining to the rehabilitative context. The framework in [8] customises full body rehabilitation within a VE through therapist support and deep learning to profile the patient and evaluate their performance in order to generate a suitable serious game. Two of the frameworks allow specialists to collect and manipulate the patient’s profile data in order to provide a suitable serious game: SIERRA [27] allows the therapist to take actions with a patient’s profile and therapeutic-related data in order to design the game interface, while ARSG [35] retrieves the patients’ personal health records through the Health Level Seven (HL7) healthcare data exchange standard, allowing the medical doctor to update the patient’s data and monitor the rehabilitation progress. Other frameworks are intended to produce more personalised serious games. The e-health framework [1] proposes adaptive hand therapy by constructing a hand disability data model (joints and motions) from the player’s health records and medical procedures with the therapists. The players’ profiles, interests and limitations are constructed to generate a personalised design via deep learning through undertaking analysis of full body exercises. The rehabilitation game model (RGM) [25] introduces a personalised exercise game by combining a set of six player types with a set of player behaviours, but this does not reflect the full range of behaviours and motivation of players’ with medical conditions as much as players in conventional games. PROGame [6] records player’s previous game actions but it does not reflect any player capabilities initially. The remaining physical rehabilitation frameworks [18, 46] do not define any player characteristics or profile.

In terms of therapeutic elements, most of the physical rehabilitation frameworks [1, 6, 8, 18, 27, 46] embed these within a game scenario and mechanics to enable the player to undergo the therapeutic tasks within the range of their capabilities. For example, the frameworks in [1, 18, 27, 46] target upper limbs, the full body rehabilitation framework [8] targets full body exercises, upper limb to lower limb (plank walk, single stance, and hand and fingers interaction), and PROGame [6] incorporates an interaction mechanism for postural control. The remaining frameworks [25, 35] do not take therapeutic elements into account and thus have no specific means to provide a specific rehabilitative context which leads to insufficient support for the particular player: RGM [25] mainly focuses on player personality towards the game, while ARSG [35] only considers muscle movement training as one of the rehabilitative goals.

The physical rehabilitation frameworks tend not to demonstrate a systematic approach to providing the substantial core gaming and motivational elements to support the player. The full body rehabilitation game [8] features only a limited number of basic core gaming and motivational elements, such as obstacles and simple geometric forms as game objects, so that the player can carry out the designated postures, and points and time limits to motivate them to continue with the exercises. PROGame [6] demonstrates player’s movements in an interaction module where the controllability occurs for the player to interact with the game object. The notable motivational element for the player is visual and auditory feedback which proved insufficient during validation, and therefore was augmented with an ability to change the appearance of the objects to use images displaying themes of particular interest to each patient. The e-health framework [1] uses a simple design interface to motivate those with mental distraction together with a visual avatar that the player may customise. Avatars are also used by other frameworks [18, 46]. RGM [25] enables certain game mechanics for the player according to the interactions or behaviours associated with their player type. Both RGM and SIERRA [27] rely on visual, auditory, or haptic feedback as motivators. The ARSG framework [35] does not incorporate any notable game features but focuses on the technology of providing an augmented reality platform.

Frameworks addressing cognitive impairment [23, 26, 54] are much more limited than those targeting physical rehabilitation, and none seek to construct a player profile to support the particular players or personalise the game experience, which limits the effectiveness of the intervention. However, all provide therapeutic elements: the intelligent pattern framework [54], which is for MCI screening, focuses on various cognitive spheres (attention, processing speed, memory, and reasoning capacity), the machine-learning based serious game framework [23] provides training for recognition, learning, memory, recall, and other executive functions, and Smart Thinker [26] supports attention and memory. In terms of core gaming and motivational elements, Smart Thinker uses feedback, score and rewards, and attention support such as ‘warning signs’ to notify the player to overcome the assigned tasks. However, the framework in [23] only provides score and levels as core gaming elements throughout the entire training.

MCI-GaTE is intended to provide a comprehensive and multidimensional set of elements so that an optimised, personalised and motivational serious game may be provided for an MCI player, thereby improving upon existing frameworks. The comparison reveals that MCI-GaTE is currently the only serious game framework that accommodates the full set of criteria for supporting personalised and gameful physical and cognitive rehabilitation. With the lack of MCI frameworks available, which at best are quite narrow or only target the screening stage, there is a need for a serious game framework such as MCI-GaTE which is specifically targeted at MCI players.

8 Concluding discussion

There is a lack of gamified solutions for MCI patients that combine physical and cognitive therapy, despite the prevalence of MCI diagnoses within the global population. Through secondary and primary research that surveyed related research literature, analysed nursing home resident profiles, and undertook in-depth interviews with a number of OTs, we established a set of themes which were organised into a serious games framework for MCI, MCI-GaTE. The framework targets players with MCI so that it may serve as an effective means for physical and cognitive rehabilitation, and thus incorporates an MCI player profile with core gaming, therapeutic, and motivational elements. A-go! is a proof-of-concept immersive serious game that was developed using the framework. It enables an MCI player to undertake physical and cognitive therapy through gestures. Given the rise in dementia among the elderly and the advantages that digital forms of rehabilitation offer, such as freedom from geographic locations, customised training programmes, and greater engagement, such frameworks and serious games are becoming vital for supporting conventional rehabilitation needs.

MCI-GaTE was developed inductively through several stages of thematic analysis which enabled successive refinements. Comparison with similar rehabilitative game frameworks confirms that it contributes a full set of criteria for supporting personalised and gameful physical and cognitive rehabilitation. Researchers and healthcare specialists may use the framework in several ways: as a set of comprehensive and established criteria by which a serious game for MCI may be evaluated, as a basis for developing frameworks for other rehabilitative domains, and as a means for proposing rehabilitative serious games for MCI patients. Our own immersive serious game, A-go!, has been designed to enable an MCI player to improve their memory capacity through repeated training supervised by an OT and demonstrates the potential of the framework in designing and implementing therapeutic experiences. Evaluation with OTs revealed that it may be more engaging and effective than traditional training tools for MCI. However, further assessment should be undertaken and the addition of enhanced features explored, which will be the basis of our future research.