1 Introduction

Acquired brain injuries (ABI) are defined as damage to the brain that can be transient or permanent, are not related to degenerative or congenital diseases, and affect brain function and structure, resulting in cognitive deficits (Cattelani et al. 2010; Gilmore et al. 2021). These can be observed through explicit difficulties in memory, language, perception, attention, and executive functions, bringing negative consequences that compromise the psychological and emotional state, as well as autonomy and independence in carrying out daily living tasks (Gilmore et al. 2021; Ponte and Fedosse 2016). The type of ABIs studied here concern traumatic brain injuries (TBI) and stroke.

Traumatic brain injury (TBI) is an intracranial type of injury caused by an external pressure commonly as a result of road accidents or other causes that affect the brain. The highest incidence of this type of ABI is present in the young adult population and typically affects the frontal lobes (Corps et al. 2015; Venkatesh and White 2016), ultimately increasing the likelihood of altered consciousness as well as memory loss and deficits in executive functions (Gracey et al. 2017; Hawryluk and Manley 2015). Faced with a moderate or severe clinical condition, the risk of developing dementia in the future increases significantly, translating into numerous disabilities in the daily life of patients (Shively et al. 2012).

Stroke on the other hand is characterized by an unexpected event, which potentiates high levels of stress that can cause physical, cognitive, social, and psychological deficits (Caprio and Sorond 2019). This type of brain injury is one of the main causes of death worldwide, and in case of survival can lead to deficits in different functions, namely, cognitive, sensory, language, motor (e.g., hemiparesis, hemiplegia, ataxia), and emotional functions (Nakling et al. 2017; Pavan et al. 2015; Silva 2010). Stroke can be classified based on its nature: haemorrhagic and/or ischemic stroke (Doria and Forgacs 2019; Kogan et al. 2020) and according to data from the National Statistics Institute of Portugal (INE), ischemic stroke is more fatal, having its highest incidence in ages over 70 years. With regard to the sequelae associated with this type of brain injury, they depend on specific factors such as the identification of the injured area, the degree of severity of the injury and its extension, as well as the risk factors of the subjects (Kogan et al. 2020; Silva 2010). Faced with diagnosis, the patient is usually confronted with several difficulties during rehabilitation. Basic abilities are usually lost that require considerable effort so that, through training, the compromised areas can be optimized and those that were not affected can be strengthened (Andrade et al. 2009; Ekechukwu et al. 2020). The origin, nature, and consequences associated with stroke can be devastating for patients as well as their family support (Pei et al. 2016).

1.1 Neuropsychological rehabilitation: virtual environments as a therapeutic tool for acquired brain injuries

Neuropsychological Rehabilitation (NR) aims to minimize as much as possible the cognitive impairment associated with the injury and is fundamentally centered on the individual as a biopsychosocial being. NR is essential in individuals with ABI to minimize the impact/cost of the lesion on the individual’s cognitive profile, and through the data collected by the neuropsychological assessment, a unique intervention program is designed for each patient in order to optimize the affected cognitive abilities (Barman et al. 2016; Ekechukwu et al. 2020).

Over the course of the years, the use of virtual environments (VE) as a therapeutic tool in rehabilitation has gained increased recognition since it is considered to offer a set of advantages over more traditional methods (e.g., such as in the case of paper and pencil tests): the patient receives immediate feedback, there are no physical consequences for the patient using screen-based visual systems, the scenario and the proposed task can be adjusted according to the needs of each individual (Gamito, et al. 2015; Shin and Kim 2015; Stevens et al. 2015) and it may be extended by including a virtual representation of the self (e.g. Soccini et al. 2022) or others like a virtual therapist (Gamito et al. 2022) to create a more natural environment for psychological intervention.

The VE scenarios are designed to reproduce the real world and everyday situations (e.g., such as activities of daily living (ADLs)) so that, in this way, there is a greater immersion on the part of the subjects, thus increasing their ecological validity (Faria et al. 2016; Gamito, et al. 2015; Parsons 2015; Rizzo and Buckwalter 1997).

Regarding the advantages of using virtual systems for rehabilitation purposes, in general, the scientific literature emphasizes: its ecological validity, since it is able to more accurately portray spatial, temporal, and situational factors in the field of application; immediate dynamic feedback; training involves increasing learning; repetition and modification of tasks appropriate to each individual; the individual is not exposed to social judgment; it is considered an effective strategy in psychotherapy; the scenarios are fully controllable as are the stimuli presented and there are no physical or mental consequences after the error (Dores et al. 2016; Faria et al. 2016; Gamito, et al. 2015; Parsons 2015). Opting for a less immersive platform (i.e., desktop virtual reality) presents a plausible way to counterbalance costs while also lowering the task effort and simulator sickness (Srivastava et al. 2019).

As specified earlier, the use of VEs offer the possibility of adjustment in accordance with the characteristics and specificities of the patient, thus favoring the course of therapy and rehabilitation. Accordingly, if the injury results from a stroke or a TBI, these individuals may suffer physical consequences preventing the practice of ADLs, in turn, with the use of these scenarios (depicting the practice of these very same activities) it is expected that through training the patient manages to overcome the difficulties in terms of ADL resulting from the injury (Faria et al. 2016; Gamito et al. 2015, 2011).

In accordance to this proposal, previous studies have shown that cognitive training of working memory/attention and memory tasks consisting of ADLs (e.g., such as shopping, finding the way to the minimarket) in highly immersive ecologically valid VEs results in improvements in overall cognitive flexibility in neurological populations such as ABIs (Faria et al. 2016; Gamito et al. 2011, 2015) and Alzheimer’s disease (Oliveira et al. 2021) as measured via traditional neuropsychological assessment tests. Additionally, important studies also suggest that training of specific ADLs and skills in VEs also promotes a transfer of gains into real-life associated activities (e.g., Cooper et al. 2021; Dobrowolski et al. 2021; Gray 2017).

1.2 Study aims and hypotheses

Taken into account the numerous advantages of VEs highlighted above, it becomes crucial to extend the number of ecologically valid platforms in VE settings to potentiate the range of cognitive skills and transfer of gains to real-life ADLs, for the most extent to the benefit of neurological populations (e.g., such as in the case of ABIs) (Cooper et al. 2021; Faria et al. 2016; Gamito et al. 2011, 2015). However, when designing a VE task (e.g., in the form of ADLs) it is important to first determine for its ecological validity and sensitivity to performance in the neurological patient before it can be rendered suitable as a tool for cognitive remediation.

A previous related study conducted by Fong et al. (2010) was carried out in groups of patients with ABI with the aim of understanding the ecological validity of using VR through a virtual automated teller machine (ATM) task, since most individuals with ABI normally lose the ability to use an ATM machine, because functions such as memory (digit memory) and executive control are largely disrupted (Fong et al. 2010; Gilmore et al. 2021). The tasks included the withdrawal of money and bank transfers. Results were consistent with the ecological validity of the ATM virtual task, since there were no significant differences in performance (measured via task execution times) between the group who conducted the virtual ATM task and the group who conducted the real ATM task (Fong et al. 2010).

The present study focuses on two central objectives, the first of which focuses on analyzing whether performance in an ATM task (more specifically, buying a train ticket at the ATM) measured via task execution times and number of errors varies depending on the type of exposure, that is, virtual or real, and to understand how the clinical condition affects the performance of the task. We intend to understand how the use of the VE task platform is feasible for cognitive training and if ecological validity is important for this purpose.

Thus, it is expected that: older participants who present lower results on neurocognitive tests (notably in the Montreal Cognitive Assessment (MoCA) and Frontal Assessment Battery (FAB) tests) perform worse on the ATM task (i.e., obtain more errors and a longer execution time), regardless of the type of task and group (hypothesis 1); and the participants referring to the ABI groups should obtain more errors and a longer time to perform/execute the task, regardless of the type of task (hypothesis 2).

2 Method

2.1 Participants

The total sample consisted of 100 participants of both genders (51 men and 49 women) with an average age of 40 years old (SD = 12, ranging between 20 and 95 years). The participants with ABI (n = 34) were recruited in a rehabilitation hospital, whereas the remaining sample (n = 66) was recruited from the general community. The inclusion criteria were being adult (equal to or above 18 years old), native Portuguese speaker, and having a diagnosis of acquired brain injury (stroke or TBI) at a sub-acute stage for the participants in the ABI group, while the control group comprised healthy participants without ABI. For participating in this study, the participants had to know the ATM by the appearance of the menus and to have previously used this equipment. Only participants with normal or corrected to normal vision were included.

The distribution of the participants by the experimental conditions was done separately for the clinical group and the non-clinical group because these groups were recruited at different stages of the study. The participants for each group were randomly distributed by either (1) Real ATM (Automated Teller Machine) or (2) Virtual ATM. The study design was between-subjects.

Most of the participants are single regarding civil status (46%) have a bachelor’s degree (49%) and have a stable job position (26%). Regarding the clinical diagnosis, ABI derived from a stroke in 24 patients and traumatic brain injury in 10 patients. The statistical comparisons between these groups were done using the Chi-square association test for categorical data and with Mann–Whitney for interval data. This analysis suggested significant differences between groups for civil status (p < 0.01), job position (p < 0.001) and age (p < 0.001). The participants from the ABI group are older, are predominantly married, and unemployed. No significant differences were found between TBI or stroke patients for gender, age, or education (p > 0.05). Table 1 shows the descriptive data for sociodemographic variables for each group.

Table 1 Sample characteristics
Full size table

2.2 Measures

The measures consisted of a brief sociodemographic form to collect the participant’s information regarding gender, age, education, civil status, and job position. The evaluation protocol comprised also four neuropsychological instruments: Montreal Cognitive Assessment (MoCA) for global cognition; Frontal Assessment Battery (FAB) for executive functions; Everyday Competence Questionnaire for independence in everyday life; Instrumental Activities of Daily Living (IADL) Scale for functionality in instrumental ADLs.

2.2.1 Montreal cognitive assessment (MoCA)

This test was used as a brief cognitive screening test that assesses visuospatial and executive functions, attention, short-term memory, language and orientation. This screening tool includes the following tasks: trail making test B, 3D cube copying, clock drawing, naming, taping, digit span backwards and forwards, serial subtraction, selective attention, sentence repetition, phonemic fluency, verbal abstraction, learning and delayed recall task, and spatial and temporal orientation. The test is administered in 10 min, the total score is 30 points, and the cut-off point is 26 with higher scores describing better cognitive function (Freitas et al. 2011; Simões et al. 2008).

2.2.2 Frontal assessment battery (FAB)

This test is a neurocognitive assessment instrument to assess the executive functions. Specifically, the six subtests of FAB assess conceptualization, mental flexibility, motor programming, sensitivity to interference, motor inhibitory control, and environmental autonomy. The total maximum score of 18 points can be divided into 3 cut-off points: less than 12 indicates dementia; from 12 to 14 indicates dysfunction. The normative value is 15–18 points (Dubois et al. 2000; Lima et al. 2000).

2.2.3 Everyday competence questionnaire (ECQ)

This scale assesses independence in everyday life for older adults where higher scores reflect greater competence. ECQ consists of 17 items, addressing housekeeping, daily routines, leisure activities, sports, manual skills, subjective well-being, and general linguistic abilities. The answers were converted into numerical scores according to an item-specific scale, and a total maximum score of 54 points could be achieved (Kalisch et al. 2011).

2.2.4 Instrumental activities of daily living scale (IADL)

This scale assesses various activities of daily living and was used to understand the degree of autonomy in everyday life activities. The IADL covers 8 domains, which includes ability to use telephone, shopping, food preparation, housekeeping, laundry, transport, ability to handle finances and responsibility for own medication. The scoring was based on three positions for each item (1 = unable, 2 = needs assistance, 3 = independent). The sum of the eight responses resulted in the total score, in which higher scores reveal higher function in daily activities (Lawton and Brody 1969).

2.3 Procedure

The project was first presented to the ethics committee of the School of Psychology and Life Sciences of Lusófona University (CEDIC) for approval. The recruitment took place in a rehabilitation hospital [Centro de Medicina e Reabilitação de Alcoitão—CMRA] in the Lisbon region of Portugal, whereas the healthy participants were recruited from both the general population in Portugal mainland and Azores. Informed consent was given to all participants. After agreeing with the informed consent, the participants filled out the sociodemographic questionnaire followed by the supervised administration of MoCA, FAB, ECQ, and IADL by a junior neuropsychologist. All these measures have no physical consequences for the participant and were completed by the entire sample. After completion, the participants were shown how to perform the task and after this training (3–5 min), they started the experimental task. The experimental task consisted of either a virtual or a real ATM replicating the same functions of the ATM available in the country in the Portuguese language. The virtual ATM application was developed in Unity 2020.3.25f1 (Unity Technologies, https://unity.com), depicting a digital version with the same screens and interaction procedure as the real ATM (Fig. 1). The screens used in this task were captured from a real ATM and were used as textures for the different menus used in this virtual ATM.

Fig. 1
figure 1

ATM task (from left to right: real ATM, virtual ATM starting screen, virtual ATM selecting train ticket, virtual ATM confirming train ticket)

Full size image

The research team including researchers and clinicians agreed on the specific purpose of the task that should not involve any monetary transaction as in the real ATM. In Portugal, ATMs can be used for different purposes, namely withdrawals, bills payment, or buying tickets. Therefore, the task consisted in consulting the timetables of trains departing from the Oriente to Pragal (which are two train stations in Lisbon region of Portugal) as they were to buy a train ticket. The task involved selecting the correct option in two different menus. The pin code for entering the account in the ATM was 0000 being the same in both conditions. In the first menu, the participants needed to select the option related to “Transportation and tolls” between 8 possible options. In the next menu, they needed to select the option for CP (train tickets) which describes the main public train company of the country. Finally, they needed to pick the name of the train station for departure and arrival. The proposed cognitive domains in this task are mainly selective and sustained attention, verbal memory and recognition memory and executive functions as planning and sequencing abilities.

The task was completed on a 15.4 inches laptop screen using a non-immersive setup. This option was chosen to offer a low-cost solution for clinical intervention, but also to lower the risk for creating simulator sickness due to exposure. The participants were instructed to use the computer mouse for interaction with the virtual ATM. In the real ATM the users had to choose the option in a touchscreen. Another difference between procedures was that the task in the real ATM was accomplished for most participants in a standing position, except for five participants in a wheelchair, whereas in the virtual ATM the participants were seated in a chair. It is important to emphasize that this task was very brief because the interaction with the ATM menus is very simple and most of the people are familiar with the ATM, so it was not predicted that the standing position would create differences in physical effort during these two conditions. The measures in this task consisted of execution time and errors (i.e., selecting an incorrect option). In the virtual ATM these measures were retrieved directly from the software that registered the selected options and the execution time. As for the real ATM, the same measures were retrieved by the experimenter through visual observation of task execution by the participants. The entire procedure considering the experimental task along with the evaluation tests had an average duration of approximately 30 min.

2.4 Data analysis

The statistical procedures were conducted in IBM SPSS Statistics 27. Given the small sample size of the groups, the statistical analyses were performed using non-parametric statistics. The dependent variables were execution times and errors for the experimental task, as well as MoCA and FAB, ECQ, and IADL scores. The independent variables were condition (virtual ATM vs. real ATM) and group (ABI vs. healthy controls). The comparisons between two independent samples were performed using the Mann–Whitney (U) test. The correlations between variables were performed using Pearson’s correlation coefficient, a parametric test that was conducted on the overall sample. The results were also further explored using the Receiver Operating Characteristics (ROC) analysis to study whether task performance distinguished groups.

3 Results

3.1 Descriptive data on neuropsychological measures and performance data

The descriptive statistics for the total scores for each of these measures are depicted in Table 2.

Table 2 Descriptive data on the neuropsychological and ATM measures
Full size table

3.2 Correlations between neuropsychological data and performance in the ATM tasks

According to Pearson correlation coefficient, age has a strong positive correlation with the time to perform the task (r = 0.601; p < 0.001), which indicates that the older the individual, the longer the time to perform the task. Regarding MoCA, FAB and ECQ, these variables were negatively and significantly correlated with age (MoCA: r = − 0.409; p < 0.001; FAB: r = − 0.316; p < 0.001; IADL: r = − 0.539; p < 0.001; ECQ: r = − 0.504; p < 0.001), indicating that as age increases, the score obtained in MoCA, FAB and ECQ decreases. Therefore, the correlations between these measures for global cognition (MoCA), executive functions (FAB) and everyday competence with performance in the ATM tasks were conducted by using partial correlations for controlling the potential age effects in these analyses. It was possible to state significant negative correlations between the scores obtained in cognitive screening measures from the MoCA and execution time (r = − 0.531; p < 0.001) and with errors (r = − 0.582; p < 0.001) while controlling for age in this analysis, as well as for the FAB total score with execution time (r = − 0.557; p < 0.001) and with errors (r = − 0.684; p < 0.001). The same pattern was observed for the functionality measures from ECQ with execution time (r = − 0.408; p < 0.001) and with errors (r = − 0.510; p < 0.001), and the IADL only with errors (r = − 0.261; p < 0.01). These results suggested that lower scores on global cognitive function, executive functions and functionality are associated with worse performance in the ATM task, irrespective of the condition real or virtual ATM.

3.3 Comparison of performance in the ATM tasks for ABI patients with healthy controls

The aim of the following analysis was to study whether performance in the ATM task is affected in ABI patients with lower cognitive function compared to healthy controls. A non-parametric analysis with the Mann–Whitney test was performed for each of the dependent variables (execution time and errors). This analysis showed that performance was significantly different from ABI patients and healthy controls both in execution time (U = 414.500; p < 0.001) and errors (U = 692.000; p < 0.001). The performance was worse in ABI patients in both measures (Figs. 2 and 3).

Fig. 2
figure 2

Histograms for execution time in ATM tasks for each group

Full size image
Fig. 3
figure 3

Histograms for errors in ATM tasks for each group

Full size image

3.4 Comparison of performance between virtual and real ATM tasks

This analysis was conducted using a Mann–Whitney test to compare performance in virtual ATM with real ATM (irrespective of the condition). The test revealed no statistically significant differences both for execution time (p > 0.05) and errors (p > 0.05) between performance in these tasks, suggesting that the type of environment in which the task was performed had no influence on this performance.

3.5 Discriminant ability of the ATM task in virtual and real ATM

We have conducted a ROC analysis to assess the sensitivity of the virtual ATM in detecting cognitive impairments associated with ABI using the task execution times and errors as test variables for the Virtual ATM and the Real ATM condition. Regarding the virtual ATM, it showed a high sensitivity for times (AUC = 0.848; SD = 0.059) and for errors (AUC = 0.824; SD = 0.074). In relation to the real ATM, it presented lower sensitivity both in terms of executive time (AUC = 0.775; SD = 0.082) and errors (AUC = 0.566; SD = 0.091). These results suggested the virtual ATM (Fig. 4) was more sensitive detecting deficits than the real ATM (Fig. 5). These results were similar when comparing this analysis between participants below/above the median age for controlling for age effects on task performance.

Fig. 4
figure 4

ROC curves for execution time and errors in virtual ATM task

Full size image
Fig. 5
figure 5

ROC curves for execution time and errors in real ATM task

Full size image

4 Discussion

The aim of this study was to validate a virtual ATM task to be used to train patients with cognitive impairments due to ABI. Our goal was to further contribute to the field of neurorehabilitation by developing a virtual reality task for the purpose of rehabilitation in future intervention studies. Therefore, we expected an effect of age, but also of cognitive and functionality in daily life activities on the performance of the ATM task. We also expected that patients with cognitive impairments due to ABI reveal worse performance in the ATM task, regardless of being the virtual or the real ATM task.

Performance in the experimental tasks was measured by task completion times (executive time) and errors in choosing the options from the ATM menus. The results suggested that older people show higher task completion times, which can be explained by the decrease in information processing speed that translates into more time spent on the task (Ebaid and Crewther 2019). The same pattern was observed for errors in the task, suggesting that aging may also influence task decisions possibly because of reduced perceptual accuracy (Starns and Ratcliff 2010).

A second outcome was that individuals with ABI diagnosis took longer times and produced more errors in performing the tasks (both virtual and real), which can be related to cognitive deficits in patients with ABI, impacting cognitive, sensory, language, and motor function (Pavan et al. 2015). In fact, the mean scores from the cognitive screening measures lie above the normative values considering age and education (Freitas et al. 2011).

It is also worth mentioning that there may be other potential confounders in these results, because ABI patients were older, and they may also have less familiarity with technology. It will be interesting for future studies to explore system usability during interaction to control for these effects (Fong et al. 2010).

The comparison of performance between virtual and real ATMs did not reveal significant differences, which agrees with findings from Fong et al. (2010). Moreover, the ability of the virtual ATM task in discriminating group membership was also explored in this study. No specific hypothesis was planned for this goal as this was only an exploratory aim. This analysis was conducted using ROC to determine whether the task discriminates the performance of the clinical group from healthy participants. In our study, the virtual ATM showed a higher discriminative power than its ‘real’ counterpart, the real ATM, in discriminating the performance of ABI patients from healthy controls. This finding suggests different working mechanisms for performing the ATM task in virtual reality or in a real machine. A previous study on virtual reality-based assessment have showed that performance of functional tasks in VR involves a broad range of cognitive processes, even in simple matching tasks resembling daily living tasks (Oliveira et al. 2021). According to Meyerbröker (2021), the underlying mechanisms by which virtual reality operates are not completely understood and are still considered a black box when it comes to virtual reality exposure.

The limitations of this study are mostly related to the sample. Due to restrictions in assessing the clinical sample it was not possible to collect a larger sample. On the other hand, part of the control group was collected in a different region of the country from the clinical group. Age was also significantly different between groups, which required to control for these effects in the analysis that age could be used as a covariate, but most procedures required the division of age as a factor limiting most statistical procedures. Another issue was the discrepancy between the number of patients with stroke and TBI, which did not allow the conduction of any direct comparison between these groups nor to study specific effects related to the lesion on task performance (Silva 2010). A potential limitation to the results may be due to the different interfaces between the virtual and the real ATM. The users interacted with the virtual ATM using the computer mouse, whereas in the real ATM interaction was done by the touchscreen from the machine. The use of the computer mouse may be more difficult specially for patients with motor disturbances than a touchscreen, but it seems that this procedure did not act as a confounder mainly because of two aspects: (1) mean execution times were not different between the virtual and real ATM; (2) data from execution times were aligned with perceptual accuracy as no differences were found for task errors.

Nonetheless, this investigation may provide a positive contribution to the field of rehabilitation since a potential tool for rehabilitation was tested. It is important to point out that the use of ATM may be an important task for cognitive training to enhance autonomy in everyday life by involving domains such as attention, memory, problem-solving, and executive functions that are often impaired in ABI (Fong et al. 2010). In fact, there are several studies suggesting the use of ecologically oriented cognitive interventions in neurological populations, such as TBI (Gamito et al. 2011), stroke (Faria et al. 2016; Gamito et al. 2015) and Alzheimer’s disease (Oliveira et al. 2021) to enhance therapeutic gains by promoting the transfer of effects into real-life activities (Cooper et al. 2021; Dobrowolski et al. 2021; Gray 2017).