Abstract
Prospective Memory (PM) is the ability to remember to do something in the future. It is often impaired after stroke and can impact on an individual’s level of independence and daily functioning. PM tasks have been criticized for their lack of ecological validity wherein test results may not be related to actual performance in daily life. With ecological validity in mind, the Virtual Reality Prospective Memory Shopping Task (VRPMST) was designed to assess two types of PM, time- and event-based. This study aimed to examine the ecological and convergent validity of the VRPMST in comparison to an experimental (Lexical Decision PM Task) and clinical measure of PM (Cambridge PM Test). Twelve individuals with stroke and 12 controls were administered three PM measures, three neuropsychological measures, and two user-friendliness questionnaires, one for the experimental PM measure and one for the VRPMST. Individuals with stroke showed impairments in PM compared to controls on all three PM measures, particularly time-based PM. Individuals with stroke were found to monitor time significantly less than controls on both the experimental PM measure and the VRPMST. The VRPMST was found to be sensitive in measuring PM, have better ecological validity when compared to the experimental PM measure, and good convergent validity. The findings of this study have helped to clarify that PM impairment does exist after stroke, possibly due to a problem in strategic monitoring. In addition, we have demonstrated how VR technology can be used to design a measure of cognitive function commonly impaired after stroke.
Similar content being viewed by others
Explore related subjects
Find the latest articles, discoveries, and news in related topics.Avoid common mistakes on your manuscript.
1 Introduction
1.1 Prospective memory
Cognitive impairments, including impairments in memory, are commonly reported after stroke (Andrews et al. 2014a). Much of the previous research investigating memory after stroke has focused on retrospective memory (RM; memory for past events and previously learned information) rather than prospective memory (PM; Andrews et al. 2014b). PM is the ability to remember to do something in the future and has been found to be impaired after stroke (Hogan et al. 2016). PM requires several interacting cognitive processes (Shum et al. 2002) with executive functions (EF) and RM playing important roles (Kant et al. 2014; Otani et al. 1997). The prefrontal lobes, in particular Brodmann Area 10, are often activated during PM tasks (Burgess et al. 2011); therefore, when this brain region and its surrounding areas are damaged due to stroke or injury, impairments in PM ability may occur. Impairments in PM can significantly impact ones’ independence and daily functioning (Man et al. 2015a) as PM failures can result in negative consequences for both individuals and those surrounding them. For example, if someone forgets to take their medication this may result in another stroke, or if an individual forgets to turn the stove off this may cause a house fire. Therefore, it is important to be able to correctly measure and identify PM impairments in order for clinicians to deliver appropriate rehabilitative techniques to help improve PM after brain injury.
Two main types of PM have been widely discussed in the literature: time- and event-based (Kvavilashvili and Ellis 1996). Time-based PM requires an individual to complete an action at a certain time or after a certain amount of time has elapsed (e.g., remembering to call your boss at 10:30 am or taking dinner out of the oven after 45 min), whereas event-based PM requires an individual to complete an action in response to a specific event that occurs in the individual’s environment (e.g., passing along a message to a family member when they return home). The multi-process framework proposes that in order for PM tasks to be completed correctly, an individual needs to use either a strategic monitoring approach to look for cues in the environment or rely on automatic spontaneous retrieval (McDaniel and Einstein 2000), with the monitoring approach dependent on the requirements of PM task requirements. It is believed that time-based PM would require greater cognitive resources for completion than event-based PM, because it depends on a strategic monitoring approach wherein an individual would continuously monitor the environment for a cue (i.e., regularly checking a clock for the time). Event-based tasks would require fewer cognitive resources for completion and usually rely on spontaneous retrieval (Einstein et al. 1995). When the cue/event that is embedded within the environment occurs (i.e., family member returning home) an individual would automatically remember that they needed to complete a PM task.
Many tools and paradigms have been designed to measure PM including experimental, naturalistic, clinical and Virtual Reality (VR), with each assessment type having their relative strengths and weaknesses. Ecological validity refers to the degree that a measure predicts daily functioning and the degree to which the cognitive demands required to complete the test resemble the cognitive demand used to complete a similar task in an individual’s everyday environment (Chaytor and Schmitter-Edgecombe 2003). In order for clinicians to reliably test one’s cognitive function, the tests used to measure these faculties must be reliable and valid, and reflect the individuals everyday performance.
While PM impairment has been reported after stroke (Hogan et al. 2016), results are mixed and dependent on the type of PM measure used or the PM type that is being measured. When PM after stroke is assessed using experimental measures (i.e., a dual-task laboratory paradigm that consists of both an ongoing and a PM task), impairments in time- but not event-based PM are often reported (Cheng et al. 2010; Kant et al. 2014). In the study conducted by Cheng et al. (2010), while completing an ongoing task, participants were required to tap the desk every 5 min for the time-based PM task, and tap the desk when they saw an animal word for the event-based PM task. Participants in the study conducted by Kant et al. (2014) were required to complete the Bourdon–Wiersma task (ongoing task) and say the word ‘regal’ aloud whenever the last configuration in a line contained three dots for the event-based PM task. For the time-based PM task, participants were instructed to insert a coin in a container after each minute has passed. While experimental PM tasks allow researchers to systematically manipulate parameters and provide good environmental control, they have been criticized for their lack of ecological validity. In addition, these tasks are usually tedious and repetitive for participants, particularly for individuals with brain injuries who may fatigue quickly. Finally, experimental measures are usually not standardized and do not have normative data, making comparisons and clinical judgments about impairment difficult.
Previous research that utilized single-item naturalistic measures generally did not find impairments in time (Kant et al. 2014) or event-based PM after stroke (Brooks et al. 2004; Kant et al. 2014; Kim et al. 2009). Kant et al. (2014) measured time-based PM by asking participants to remind the researcher to make a phone call after 30 min. Brooks et al. (2004), Kant et al. (2014), and Kim et al. (2009) utilized the Remember a Belonging Task from the Rivermead Behavioral Memory Test (RMBT; Wilson et al. 1985) to measure event-based PM. In this task participants have to give a personal item (i.e., a watch) to the researcher at the beginning of a testing session and are instructed to ask for the item back when the session concludes. One exception to these findings was reported by Brooks et al. (2004) who used an additional naturalistic measure of event-based PM wherein participants were required to ask for a written explanation of the study at the end of another task. On this task, individuals with stroke performed significantly poorer than healthy controls. While one would think that the best way to measure PM would be to get participants to complete real-life PM tasks, naturalistic tasks are often single-item measures which may not be robust enough to reliably measure PM alone (Kim et al. 2009). In addition, it is difficult for experimenters to control and manipulate the task parameters of naturalistic tasks. Furthermore, it may be hard to reliably measure performance for any tasks that are not completed in the presence of the experimenter as individuals may rely on other people or aids like a calendar alarm to remind them to complete the task.
A few psychological tests have been developed, standardized, and psychometrically validated to assess PM in clinical settings. When using the psychometrically validated Cambridge PM test (CAMPROMPT; Wilson et al. 2005) individuals with stroke were found to perform significantly poorer than controls on both time- and event-based PM (Barr 2011; Hogan et al. in press; Man et al. 2015b). The CAMPROMPT contains six different PM tasks (3 event- and 3 time-based) which are to be completed during or at the end of an ongoing paper and pen task. Research using another clinical measure, the Virtual Week which sees participants complete a number of time- and event-based PM tasks in response to cues provided by a game board (Rendell and Henry 2009), found that individuals with stroke showed impairments on time-based PM only (Kim et al. 2009). While clinical measures were developed with ecological validity in mind, the tasks that participants are required to perform in the majority of PM measures are still not typical of tasks that individuals are required to complete in their daily lives, resulting in the ecological validity being questioned (Canty et al. 2014).
While the results are mixed, the majority of the research suggests that PM exists after stroke, particularly for time-based PM. As previously outlined, time-based PM tasks require more cognitive resources for completion than event-based PM due to the type of monitoring required for task completion. Kant et al. (2014) found that individuals with stroke and controls who checked the time more frequently performed better on time-based PM, but individuals with stroke checked the time significantly less than controls, leading to poorer time-based PM performance. Therefore, deficits in monitoring may lead to PM impairment after stroke.
1.2 Virtual reality
Due to the limitations of current measures of neuropsychological function, the push for more ecologically valid measures, and advances in technology, VR has garnered considerable attention in recent years, offering possible solutions to these assessment problems (Knight and Titov 2009). VR comprises artificial computer-generated environments containing distinctive sensory properties and allows users to interact with the virtual environment in real-time (Mitrovic et al. 2016). VR can benefit neuropsychological assessment in many ways, for example, it can balance the demands of high ecological validity along with the requirements of sensitivity and specificity. VR platforms allow the embedding of naturalistic everyday PM tasks into environments that participants are familiar with, therefore, helping to overcome the practical and reliability issues that arise when assessing PM in real-life scenarios. Additionally, VR platforms provide the required conditions for standardized testing to take place, including a consistent environment, and potentially unlimited repetitions of a single assessment. At the same time, VR offers the flexibility to adjust specific aspects of a task, like complexity, difficulty, or response requirements for individuals, adjusting for possible impairments. Due to their novelty, VR environments are also believed to enhance test-takers’ motivation and interest (Sweeney et al. 2010; Canty et al. 2014) compared to tedious and repetitive laboratory tasks. Lastly, VR environments are particularly suited to assessing PM, because they can construct and facilitate a standardized procedure with ease but can also assess the dynamic coordination of multiple cognitive abilities simultaneously (Knight and Titov 2009).
Utilizing VR in the assessment of PM is an emerging field and has been used in multiple populations, including traumatic and acquired brain injury (TBI/ABI; Banville and Nolin 2012; Canty et al. 2014; Kinsella et al. 2009; Morris et al. 2002; Sweeney et al. 2010; Yip and Man 2013), stroke (Brooks et al. 2004; Mitrovic et al. 2016), schizophrenia (Kurtz et al. 2006; Man et al. 2018), aging (Debarnot et al. 2015), and healthy populations (Gonneaud et al. 2014). Many virtual environments have been developed, including shopping centers (Canty et al. 2014; Kinsella et al. 2009; Yip and Man 2013), apartments/houses (Banville and Nolin 2012; Brooks et al. 2004; Mitrovic et al. 2014; Rose et al. 1999; Sweeney et al. 2010), and cities/towns (Debarnot et al. 2015; Gonneaud et al. 2014). The literature has shown that VR platforms are suitable for use with clinical populations and can be adapted to suit the intrinsic needs of researchers and clinical populations.
1.3 Using VR to investigate PM after stroke
VR research investigating PM after stroke is limited with one study using a VR platform to assess PM (Brooks et al. 2004) and another using a VR platform in PM rehabilitation wherein participants could practice their newly learnt strategies to aid in prospective remembering. Brooks et al. (2004) compared PM performance between 25 individuals with stroke and 25 age-matched controls. The virtual environment consisted of a four-room bungalow wherein participants were instructed to help the owner pack to move house. In addition, they completed a series of event- (put ‘Fragile’ notes on items with glass components before moving them), activity- (close the door every time you leave the kitchen to keep the cat in), and time-based PM tasks (click on the red button in the hall next to the clock every 5 min to let the removalists into the house). It is important to note that participants did not control the virtual environment themselves. Rather the participant would tell the researchers what to do and the researcher would control the simulation. Therefore, it could be argued that the immersive feature of VR could be lost in the study. Brooks et al. found that the controls performed better than individuals with stroke on all PM tasks; however, significant differences were only found for event- and activity-based PM, whereas time-based PM was only marginally significantly different (p = 0.05). Moreover, the VR task was deemed to be more sensitive to PM impairments than the Remember a Belonging Task (Wilson et al. 1985).
It is important that standardized and ecologically valid PM measures, including VR measures, continue to be developed for use in both experimental and clinical settings. PM measures should aim to incorporate a variety of real-life tasks, naturalistic environments, platforms that engage participant’s attention, increase motivation, and feature user-friendly controls. These assessments should not only focus on the measurement of PM but also be able to manipulate and control certain factors to determine the underlying reasons or mechanisms of PM impairments.
1.4 Aims and hypotheses
This study aimed to design a VR task that measures both event- and time-based PM as well as time-monitoring (viz., Virtual Reality Prospective Memory Shopping Task; VRPMST) and use it to compare individuals with stroke and healthy controls. The study also aimed to compare the VRPMST to a psychometric (viz., CAMPROMPT), and experimental PM task (viz., Lexical Decision PM Task; LDPMT). Additionally, the VRPMST’s ecological validity in comparison to an experimental PM measure (viz., LDPMT) would be studied. Lastly, the study aimed to evaluate the convergent validity of the VRPMST by investigating the relationships between PM performance on the VRPMST PM performance and three standardized neuropsychological and two other PM measures.
H1: It was hypothesized that individuals with stroke would perform poorer than controls on both time- and event-based PM on all PM measures.
H2: Individuals with stroke would monitor the time significantly less than controls on both the LDPMT and VRPMST and that monitoring would significantly correlate with time-based PM performance.
H3: It was predicted that the VRPMST would be just as sensitive in measuring PM as the CAMPROMPT and LDPMT and have high ecological validity wherein both individuals with stroke and controls would rate the VRPMST more favorably than the LDPMT in terms of subjective user-friendliness.
H4: Lastly, it was hypothesized that the VRPMST would have strong convergent validity resulting in strong correlations between the VRPMST, neuropsychological measures, and the other PM measures (LDPMT and CAMPROMPT).
2 Method
2.1 Participants
Twelve individuals with stroke (6 females) were recruited through multiple stroke organizations and health facilities in South-East Queensland, Australia. Inclusion criteria included: between 18 and 85 years old, diagnosed cerebrovascular accident within the last five years, and living in the community at least one-month post-stroke. Exclusion criteria included: stroke located in cerebellum/brainstem, history of previous brain injury/neurological illness other than stroke, diagnosed dementia/neurodegenerative illness; significant psychiatric disorder; insufficient communicative ability (i.e., severe aphasia); significant visual/hearing/hand function impairment that would impact on participation ability; or current alcohol/substance abuse. On the Modified Rankin Scale (MRS; Rankin 1957) 3 individuals had no significant disability, 4 had slight disability, 4 moderate, and 1 moderately severe disability. On the Nottingham Extended Activities of Daily Living (NEADL; Nouri and Lincoln 1987) questionnaire, participants reported that they completed most activities of daily living independently or independently with difficulty (M = 53.92, SD = 13.98). Specific demographic stroke information is displayed in Table 1. Twelve control participants (9 females) aged between 18 and 85 years, with no history of neurological illness, brain injury, or stroke, were recruited through Griffith University (subject-pool, Learning Space, and research volunteer email) and various community groups (i.e., seniors’ groups). Exclusion criteria included: significant psychiatric illness, current alcohol/substance abuse, significant visual/hearing impairment, or insufficient communicative ability. The two groups did not differ on any demographic variable (Table 1).
2.2 Measures
2.2.1 VRPMST
The VRPMST, developed for the current study, was adapted from the VR framework (shopping center) used by Canty et al. (2015). The VRPMST features a dual-task paradigm (ongoing and event- and time-based PM tasks) run on a laptop, with an iPad used to wirelessly control the simulation (Fig. 1). The virtual environment was developed and adapted by a team of Computer Graphic Artists and Programmers at VRspace Pty Ltd. The digital assets (i.e., characters and shopping center) were created using Auto-desk 3D Studio Max and the functionality of the application was developed using Unity3D game engine. Participants completed the task on a Dell M6700 note-book computer. Hardware requirements are Windows 7, 3.0 GHz CPU, a dedicated Graphics Card/an adaptor with at least 2 Mb texture memory and 4 Gb RAM. In addition, a wireless router connects the iPad which is used to control the simulation from an app built by the programmers to the laptop via Wi-Fi. The laptop is placed in front of the participant and adjusted so that they can comfortably view the screen.
A practice trial (5 min) where participants can become familiar with the controls (locate and view the map, list, and watch) and complete an example ongoing task (i.e., enter a shop and complete a task), as well as receive confirmation of instructions are completed before starting the assessment, ensuring that participants understand the task requirements and are familiar with the navigation and response controls. The ongoing task (12 errands) is completed by moving around the virtual shopping center and entering stores in a set order. The tasks are designed to mimic everyday activities completed at a shopping center (e.g., buy a birthday card from the newsagent). To facilitate immersion, animated advertisement boards and background chatter are built into the ongoing task. For time-based PM, participants are told that their doctor has instructed them to keep track of their heart rate; therefore, they need to check their heart rate (press H on the iPad) every 3 min. Participants can check their virtual watch (elapsed time appears for three seconds; Fig. 2) by pressing the TIME button on the iPad. Time-based PM is scored as deviation in response to the target time (i.e., 3 points = ≤ 15 s after target time, 2 points = > 15 but ≤ 30 s, 1 point = > 30 but ≤ 60 s, 0 points = > 60 s after specified time or no response at all) with similar methods used in previous PM research (Canty et al. 2014; Einstein et al. 1995). Monitoring is measured as the overall number of clock checks.
The VRPMST features two types of event-based PM tasks. Firstly, participants are told that they are keeping track of food expenditure. Therefore, after purchasing food items (4 items), they are required to collect a receipt (by pressing R on the iPad; Fig. 3). Secondly, participants are informed that they lost their glasses on their previous shopping trip. Therefore, they need to ask the center’s security guard every time they see one (4 times) whether they have found their glasses (press S on the iPad; Fig. 4). Event-based PM is scored out of 8, with higher scores indicating better performance.
2.2.2 CAMPROMPT
The CAMPROMPT (Wilson et al. 2005) is a standardized psychometric PM measure commonly used in experimental and clinical settings and was used in the current study to obtain a baseline measure of PM. It features three event- and three time-based PM tasks to be completed throughout or at the end of the ongoing paper puzzle task. The CAMPROMPT has a parallel form, excellent inter-rater reliability (r = 0.99), test–retest reliability (Kendall's tau-b = 0.64) and has been used in previous stroke research (Barr 2011; Man et al. 2015b).
2.2.3 LDPMT
Developed for the current study and adapted from Canty et al. (2014) and Maujean et al. (2003), the LDPMT features a dual-task paradigm run on an iPad. Event-based PM is assessed first followed by a break period to avoid fatigue, then time-based PM, with practice trials and confirmation of instructions before each section. This ensures participants' understanding of the task requirements and familiarity with controls. Section one features an ongoing Lexical Decision Task (LDT) where participants are to determine whether the text on the screen is an English word or a non-word. Participants are shown a series of 55 English words sourced from the MRC Psycholinguistic Database (Wilson 1988), and 44 legal non-words that follow English language rules sourced from the ARC Nonword Database (Rastle et al. 2002), and 6 event-based PM animal words in a pre-set random order. If the text on the screen is an English word (e.g., GIRL) participants are to press the left button on the iPad. If the text is a non-word (e.g., YIMS), they are to press the middle button and if the text is an animal word (e.g., FROG) they are to press the button on the right. Each item is four letters long, presented in capital letters in a clear legible font and displayed for 3 s then disappears; however, participants can take as long as needed to respond. After responding, a plus sign (+) is displayed center-screen for 1 s, followed by the next trial. The ongoing LDT is scored as percentage correct (number of correct responses divided by 99 trials, multiplied by 100), and event-based PM is scored out of six, with higher scores indicating better PM performance.
For time-based PM, participants complete another LDT (i.e., determine whether the text on the screen is a word or a non-word, but, unlike the event-based task, no animal words are included) wherein they are required to press the button on the right every 60 s. Six time-based PM trials are placed throughout the task (i.e., 1 min, 2 min…etc.) and the assessment automatically ends at 6 min and 30 s. To check the elapsed time, participants can press a gray bar at the top of the screen as often as they like. The time is presented for 2 s then disappears. LDT percentage correct score is calculated (number of correct responses divided by total trials completed, multiplied by 100), and time-based PM is scored out of 18, with higher scores indicating better performance. Time-based PM is scored as deviation in response to the target time (i.e., 3 points = ≤ 5 s after target time, 2 points = > 5 but ≤ 10 s, 1 point = > 10 but ≤ 15 s, 0 points = > 15 s after specified time or no response at all). While LDT tasks have not been previously used in PM assessment post-stroke, they have been widely used in the experimental PM literature (Einstein et al. 1995; Ellis and Milne 1996) and in clinical populations like TBI (Canty et al. 2014; Maujean et al. 2003); therefore, they are a good criterion to validate the VRPMST against.
2.2.4 User-friendliness scale (UFS)
The UFS-LDPMT (eight items) and UFS-VRSPMT (9 items) were adapted from Canty et al. (2014) to assess participants’ subjective testing experience of the LDPMT and VRPMST using a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree). The UFS assesses to what degree the measures reflected everyday activities, was interesting, clear and easy to learn, task difficulty, and if they would recommend the task.
2.2.5 Neuropsychological measures
The participants completed a number of neuropsychological tests commonly used in clinical and experimental settings to assess executive function (Trail Making Task; TMT; EF component = Part B minus Part A; Reitan 1992), verbal learning and memory (Hopkins Verbal Learning Test-Revised; HVLT-R; Benedict et al. 1998), and global cognitive function (Montreal Cognitive Assessment Scale; MoCA; Nasreddine et al. 2005) that covers attention and concentration, EF, memory, language, visuoconstructional skills, conceptual thinking, calculations, and orientation. These measures were selected to provide evidence of convergent validity as the processes they assess are related to PM (Canty et al. 2014; Fleming et al. 2008; Kant et al. 2014; Martin et al. 2003) and have previously been used in stroke populations (Andrews et al. 2014b; Cumming et al. 2011; Kant et al. 2014).
2.3 Procedure
Prior to study commencement, ethical clearance was obtained from the relevant hospital and university ethics committees and written informed consent was obtained from all participants before the first session. An assessment battery was conducted over two sessions, each taking approximately 1.5–2 h with breaks if required to avoid fatigue. Assessment sessions were held in the participants’ homes or in a quiet room at the university. As testing took place in an office or in individuals’ homes, the testing conditions were not the same for each participant; however, it was ensured that there was adequate lighting and that each testing session was free from excess noise/distractions. After completion of both sessions, participants were compensated $20 or 2 credit points if recruited through the university’s subject-pool.
2.4 Data analysis
The data were analyzed using IBM SPSS Statistics for Windows (Version 25; IBM Corporation 2017). The data were screened for accuracy, missing values, outliers, and normality. Descriptive statistics were obtained for basic demographic data and are outlined in the participants section. Comparisons between the stroke and control groups on the VRPMST, CAMPROMPT, and LDPMT were made using independent-samples one-tailed t tests. Effect sizes were determined using Cohen’s d: small = 0.2, medium = 0.5, and large = 0.8 (Hu 2010). Independent samples t tests were used to analyze the results between groups on the UFS-LDPMT, and UFS-VRPMST. Paired-samples t tests were used to compare the LDPMT and VRPMST on user-friendliness. Pearson’s correlational analyses were used to examine the relationships between the VRPMST and other PM measures (CAMPROMPT and LDPMT) and neuropsychological measures (TMT, HVLT-R, and MoCA). Effect sizes for correlations were interpreted as: small = 0.1, moderate = 0.3, and strong = 0.5 (Cohen 1998).
3 Results
Due to technical errors, one control LDPMT and one stroke VRPMST data point were missing. One individual with stroke had missing data for time-based LDPMT due to fatigue-related noncompletion. The VRPMST ongoing (ceiling effect for controls), LDPMT event-based (stroke), UFS-VRPMST item 5 (ongoing task; ceiling effect for controls), and UFS-VRPMST item 9 (would recommend to others; stroke) variables were significantly skewed. Transformations were applied; however this did not change the results; therefore, the raw data is reported. One outlier (control) for item 4 (task easy to learn) of the UFS-VRPMST, one outlier (control) on the TMT, and two outliers (controls) on LDPMT monitoring were removed from the dataset. Two outliers (control and stroke) were found on item 6 (found event-based PM difficult) on the UFS-LDPMT; however they did not impact on the results; therefore the raw data is reported. Table 2 summarizes the means, standard deviations, and t tests between groups on all PM and neuropsychological measures.
3.1 PM performance between groups
3.1.1 VRPMST
Both groups scored similarly on the ongoing task, indicating that they did not find the task too difficult. Controls performed significantly better on time-based PM compared to individuals with stroke (medium effect size). Additionally, controls monitored the time significantly more than individuals with stroke (large effect size). Controls scored higher than individuals with stroke on event-based PM; however, no significant difference was found. VRPMST monitoring significantly correlated with VRPMST time-based PM for both groups (stroke: r = 0.86, p < 0.001; control: r = 0.70, p = 0.012).
3.1.2 CAMPROMPT
Individuals with stroke performed significantly worse on the CAMPROMPT compared to controls on both event- and time-based PM (large effect sizes).
3.1.3 LDPMT
Both groups scored similarly on the ongoing tasks of the LDPMT (p > 0.05 for both event- and time-based LDTs), scoring above 89% correct. The stroke group performed poorer than controls on event-based PM; however the difference was not significant (medium effect size). Controls performed significantly better than individuals with stroke on time-based PM (large effect size) and were found to monitor significantly more than individuals with stroke (large effect size). Monitoring on the LDPMT was found to significantly positively correlate with time-based PM for individuals with stroke (r = 0.66, p = 0.029; strong) but not significantly correlate for controls on time-based PM (r = -0.25, p = 0.52; moderate).
3.2 User friendliness
User-friendliness was examined for the LDPMT and VRPMST (Table 3). Ease of understanding on the LDPMT was significantly different between groups with controls finding it easier to understand; however the mean scores for both groups represented agree. Ease of learning on the LDPMT was significantly different between the groups, with stroke reporting a neutral to agree response compared to the controls agree to strongly agree response; therefore the results are still favorable for the LDPMT. Individuals with stroke reported a neutral to agree response when recommending the LDPMT, significantly different from the controls response of agree to strongly agree. Controls reported that the VRPMST included everyday activities and was easy to understand significantly more than individuals with stroke; however both groups mean responses ranged between agree and strongly agree for both items. To examine the differences between the subjective ratings of the LDPMT and VRPMST, paired-samples t tests were conducted (Table 4). Both groups rated the VRPMST to contain significantly more activities that were similar to everyday life compared to the LDPMT.
3.3 Correlational analyses
Individuals with stroke performed significantly worse than controls on the TMT (large effect size) and both HVLT-R scores (large effect sizes; Table 2). No significant differences were found between the groups on the MoCA. Correlational analyses were conducted to investigate the relationships between performance on the VRPMST, other PM measures, and neuropsychological measures (see Table 5). Time- and event-based VRPMST scores were significantly strongly correlated with both time- and event-based PM on the CAMPROMPT for the stroke sample. For controls, only event-based VRPMST significantly strongly correlated with both time- and event-based PM on the CAMPROMPT. Significant strong correlations were found between VRPMST time-based PM and LDPMT event- and time-based PM for the stroke sample but not for the control group. For individuals with stroke event-based PM on the VRPMST was significantly strongly correlated with the TMT, both HVLT-R scores, and MoCA, while time-based PM was significantly strongly correlated to the TMT, HVLT-R total recall score, and the MoCA. For controls VRPMST event- and time-based PM was significantly strongly correlated to TMT; however only the HVLT-R percent retained score was significantly correlated with time-based PM.
4 Discussion
This study aimed to design a PM measure using VR to assess individuals with stroke and healthy controls and compare the results to a standardized and an experimental measure of PM. Hypothesis one, that individuals with stroke would perform poorer than controls on time- and event-based PM on all PM measures, was partly supported. Consistent with previous research (Barr 2011; Hogan et al. 2020; Man et al. 2015b) individuals with stroke performed significantly poorer than controls on both event- and time-based PM on the CAMPROMPT. On the VRPMST and LDPMT individuals with stroke performed significantly poorer than controls on time-based PM. While individuals with stroke performed worse than controls on event-based PM on both the LDPMT and VRPMST, no significant differences were found between the groups.
The results of the current study suggest that PM impairment does exist after stroke, particularly for time-based PM, which is in line with previous research (Hogan et al. 2016). While previous literature has not utilized LDTs in the assessment of PM after stroke, similar studies using dual-task paradigms have been conducted. The current findings are consistent with those of Cheng et al. (2010) and Kant et al. (2014) who found that individuals with stroke performed significantly poorer on time- but not event-based PM compared to controls when assessed with experimental PM measures. These findings are most likely due to the difference in measure type and PM task type, as event-based PM is believed to require fewer cognitive resources for completion than time-based PM (Einstein et al. 1995). This is because event-based PM tasks have more environmental cues than time-based tasks, as such they require less self-initiation. The general consensus is that time-based tasks demand more resources than event-based; however, it is possible that event-based PM tasks could be equal to or require more resources than time-based tasks dependent on the task itself and the availability of PM cues within the environment. However, for the current study, the subjective ratings by both group of participants suggest that time-based tasks were more difficult than the event-based PM tasks.
Only one previous study has used VR to assess PM in stroke populations (Brooks et al. 2004). While the previous research found significant differences between individuals with stroke and controls in event-based PM and a marginal difference (p = 0.05) in time-based PM, the current study found differences in time-based PM but not event-based PM. One possible explanation for inconsistent findings may be the difference in cue saliency or ability to monitor the time between the two different PM measures. The TIME button was clearly displayed on the iPad in the current study. However, in the study conducted by Brooks et al., to check the time, participants had to ask the experimenter to enter the hall and press a red button next to a digital clock. Therefore, in the current study that participants may have been prompted to check the clock more often than in the Brooks study as the cue was in plain sight at all times while completing the test. It could be suggested that both individuals with stroke and controls in the Brooks study found the time-based PM task harder due to the clock not being in direct view at all times. The hypothesis that individuals with stroke would monitor the time significantly less than controls on both the LDPMT and VRPMST was supported. Additionally, time monitoring was found to significantly correlate with time-based PM on both the LDPMT and VRPMST for individuals with stroke, with greater monitoring resulting in higher PM performance. These results suggest that PM impairment after stroke may be due to deficits in strategic monitoring ability, which would negatively impact time-based PM, and possibly event-based PM tasks that lack salient cues.
The current study also aimed to corroborate the ecological validity of the VRPMST in comparison to a conventional experimental PM measure (LDPMT). The hypothesis that both individuals with stroke and controls would rate the user-friendliness of the VRPMST more favorably than the LDPMT was supported. Overall, both measures were rated favorably by both groups. Participants agreed that the tasks were interesting, instructions were clear and easy to understand, tasks were easy to learn, and that they would recommend the tasks to other people. Individuals with stroke reported that they would recommend the VRPMST just as much as the LDPMT. This finding suggests that VR is suitable to use in stroke populations and positively enhanced the participants’ testing experience. The most important finding was that both groups deemed the VRPMST to contain more tasks that were similar to everyday life than the LDPMT, suggesting higher ecological validity. The VRPMST was also found to be sensitive to the assessment of PM, particularly time-based PM, as the results were similar to those found on the CAMPROMPT and LDPMT. As the VR measure was rated favorably and similar to the experimental PM measure, and individuals with stroke were able to control and engage with the virtual environment, VR measures should continue to be developed and used in stroke populations.
On both the LDPMT and VRPMST, the groups did not find the ongoing task difficult, evidenced by UFS self-reports and high scores, accompanied by no significant difference between groups when questioned about task difficulty. These results suggest that the ongoing task was unlikely to impact on PM performance as they did not require high levels of cognitive demand. On both the LDPMT and VRPMST, both groups reported the time-based PM tasks to be more difficult than the ongoing and event-based PM tasks. Einstein et al. (1995) suggested that time-based PM is harder to complete most likely due to participants needing to use a strategic monitoring approach to continually monitor the time, further suggesting that time-based PM may be impaired due to monitoring deficits after stroke. Overall both groups reported that they did not find any of the PM tasks overly difficult (strongly disagree to neutral).
The last objective of the study was to evaluate the convergent validity of the VRPMST by exploring its relationships with neuropsychological and other PM measures. The study found evidence of convergent validity for the VRPMST and partially supported the final hypothesis that stated that the VRPMST would have strong correlations with neuropsychological measures and other PM measures. For controls, only time-based PM was significantly correlated with a measure of RM while both event- and time-based PM was significantly correlated with EF. For individuals with stroke both event and time-based PM on the VRPMST was significantly correlated with a measure of RM, EF, and global cognitive function. This is consistent with previous literature that suggests that RM and EF are related to experimental time-based PM and RM to event-based PM after stroke (Kant et al. 2014). Within TBI samples, previous research has found EF to be predictive of event-based PM (VRST and CAMPROMPT; Canty et al. 2014; Fleming et al. 2008, respectively) and EF and RM to be related to total PM performance (combination of event- and time-based PM measured by VRST; Canty et al. 2014). The current findings further support the notion that executive processes and intact RM are required for PM completion (Fish et al. 2007; Kliegel et al. 2004; Kvavilashvili et al. 2009). Further preliminary evidence of the convergent validity was suggested by strong significant correlations observed between the event- and time-based PM scores on the VRPMST and LDPMT for individuals with stroke. LDT tasks have been used as a benchmark tool in the experimental PM literature (Einstein and McDaniel 1990; Ellis and Milne 1996); therefore finding relationships between the VRPMST and the LDPMT provides preliminary support for the convergent validity of the VRPMST. Further evidence of the convergent validity of the VRPMST was suggested by the strong significant correlations to the CAMPROMPT, a psychometric measure of PM, for individuals with stroke.
Overall impairments in both time- and event-based PM were found for individuals with stroke compared to controls, suggesting impairments in PM after stroke. The current findings add to the little research that has been conducted focusing on PM after stroke, highlighting potential impairments and that more research needs to be conducted to determine whether PM impairment exists after stroke and what are the possible reasons for impairment. This will enable clinicians and researchers to assist in the rehabilitation of PM after stroke. The small sample size is a limitation; therefore, the current study may be underpowered to find significant results. Future research should aim to recruit larger numbers of individuals with stroke and age and education matched controls in order to complete more sophisticated statistical analyses which may provide more concrete conclusions. A larger sample size would also allow researchers to control for certain confounding or predictive factors like age, severity of stroke, or location of brain injury. In addition, both the LDPMT and VRPMST experienced some minor technical errors resulting in some missing data. While the current research has its limitations, it has helped to clarify the findings of previous research in that PM impairment exists after stroke, particularly time-based PM (Barr 2011; Cheng et al. 2010; Kant et al. 2014; Man et al. 2015b). Impairments may be due to deficits in strategic monitoring ability. While monitoring was easy to measure in the current study for time-based PM (i.e., total number of clock checks) future research should aim to examine monitoring strategies in both time- and event-based PM, possibly through the use of eye-tracking technology. The findings also suggest that VR can be used to assess PM in both clinical and healthy populations and that the VRPMST is sensitive in measuring PM. The VRPMST was also found to be an ecologically valid tool when compared to the LDPMT and have good convergent validity when assessed against neuropsychological tests and the CAMPROMPT, respectively. Given the benefits of using VR to assess PM and the favorable ratings provided by both groups in the current study, it is suggested that future research continues to develop and improve the paradigms to accurately and effectively measure PM after stroke and in other clinical populations.
References
Andrews G, Halford GS, Chappell M, Maujean A, Shum DH (2014a) Planning following stroke: a relational complexity approach using the tower of London. Front Hum Neurosci 8:1–14. https://doi.org/10.3389/fnhum.2014.01032
Andrews G, Halford GS, Shum DH, Maujean A, Chappell M, Birney DP (2014b) Verbal learning and memory following stroke. Brain Inj 28:442–447. https://doi.org/10.3109/02699052.2014.888758
Banville F, Nolin P (2012) Using virtual reality to assess prospective memory and executive functions after traumatic brain injury. J Cybertherapy Rehabil 5:45–55
Barr AC (2011) Prospective memory functioning after stroke: a research portfolio. Dissertation, University of Edinburgh
Benedict RHB, Schretlen D, Groninger L, Brandt J (1998) Hopkins verbal learning test - revised: normative data and analysis of inter-form and test-retest reliability. Clin Neuropsychol 12:43–55. https://doi.org/10.1076/clin.12.1.43.1726
Brooks BM, Rose FD, Potter J, Jayawardena S, Morling A (2004) Assessing stroke patients’ prospective memory using virtual reality. Brain Inj 18:391–401. https://doi.org/10.1080/02699050310001619855
Burgess PW, Gonen-Yaacovi G, Volle E (2011) Functional neuroimaging studies of prospective memory: what have we learnt so far? Neuropsychologia 49:2246–2257. https://doi.org/10.1016/j.neuropsychologia.2011.02.014
Canty AL, Fleming J, Patterson F, Green HJ, Man D, Shum D (2014) Evaluation of a virtual reality prospective memory task for use with individuals with severe traumatic brain injury. Neuropsychol Rehabil 24:238–265. https://doi.org/10.1080/09602011.2014.881746
Canty AL, Neumann DL, Fleming J, Shum DHK (2015) Evaluation of a newly developed measure of theory of mind: the virtual assessment of mentalizing ability. Neuropsychol Rehabil 27:834–870. https://doi.org/10.1080/09602011.2015.1052820
Chaytor N, Schmitter-Edgecombe M (2003) The ecological validity of neuropsychological tests. Neuropsychol Rev 13:181–197. https://doi.org/10.1023/B:NERV.0000009483.91468.fb
Cheng H, Tian Y, Hu P, Wang J, Wang K (2010) Time-based prospective memory impairment in patients with thalamic stroke. Behav Neurosci 124:152–158. https://doi.org/10.1037/a0018306
Cohen J (1988) Statistical power analysis for the behavioural sciences. Lawrence Erlbaum, New Jersey
Cumming TB, Bernhardt J, Linden T (2011) The Montreal cognitive assessment: short cognitive evaluation in a large stroke trial. Stroke 42:2642–2644. https://doi.org/10.1161/strokeaha.111.619486
Debarnot U, Crépon B, Orriols E, Abram M, Charron S, Lion S, Roca P, Oppenheim C, Gueguen B, Ergis AM, Baron JC, Piolino P (2015) Intermittent theta burst stimulation over left BA10 enhances virtual reality-based prospective memory in healthy aged subjects. Neurobiol Aging 36:2360–2369. https://doi.org/10.1016/j.neurobiolaging.2015.05.001
Einstein GO, McDaniel MA (1990) Normal aging and prospective memory. J Exp Psychol Learn Mem Cogn 16:717–726. https://doi.org/10.1037/0278-7393.16.4.717
Einstein GO, McDaniel MA, Richardson SL, Guynn MJ, Cunfer AR (1995) Aging and prospective memory: examining the influences of self-initiated retrieval processes. J Exp Psychol Learn Mem Cogn 21:996–1007. https://doi.org/10.1037/0278-7393.21.4.996
Ellis J, Milne A (1996) Retrieval cue specificity and the realization of delayed intentions. Q J Exp Psychol Sect A Hum Exp Psychol 49A:862–887. https://doi.org/10.1080/027249896392333
Fish J, Evans JJ, Nimmo M, Martin E, Kersel D, Bateman A, Wilson BA, Manly T (2007) Rehabilitation of executive dysfunction following brain injury: “content-free” cueing improves everyday prospective memory performance. Neuropsychologia 45:1318–1330. https://doi.org/10.1016/j.neuropsychologia.2006.09.015
Fleming J, Riley L, Gill H, Gullo MJ, Strong J, Shum D (2008) Predictors of prospective memory in adults with traumatic brain injury. J Int Neuropsychol Soc 14:823–831. https://doi.org/10.1017/S1355617708080971
Gonneaud J, Piolino P, Lecouvey G, Madeleine S, Orriols E, Fleury P, Eustache F, Desgranges B (2014) Assessing prospective memory in young healthy adults using virtual reality. Int J Child Health Hum Dev 7:405–413
Hogan C, Cornwell P, Fleming J, Shum D (2020) The assessment and prediction of prospective memory after stroke. J Int Neuropsychol Soc 26:873–882. https://doi.org/10.1017/S1355617720000405
Hogan C, Fleming J, Cornwell P, Shum D (2016) Prospective memory after stroke: a scoping review. Brain Impair 17:123–142. https://doi.org/10.1017/BrImp.2016.12
Hu Q (2010) Effect size, measures of. In: Salkind NJ (ed) Encyclopedia of research design. SAGE Publications Inc, Thousand Oaks CA, pp 408–412
Corporation IBM (2017) IBM SPSS statistics (Version 25). IBM Corporation, New York
Kant N, van den Berg E, van Zandvoort MJE, Frijns CJM, Kapelle LJ, Postma A (2014) Functional correlates of prospective memory in stroke. Neuropsychologia 30:77–83. https://doi.org/10.1016/j.neuropsychologia.2014.05.015
Kim HJ, Craik F, Luo L, Ween JE (2009) Impairments in prospective and retrospective memory following stroke. Neurocase 15:145–156. https://doi.org/10.1080/13554790802709039
Kinsella GJ, Ong B, Tucker J (2009) Traumatic brain injury and prospective memory in a virtual shopping trip task: does it matter who generates the prospective memory target? Brain Impair 10:45–51. https://doi.org/10.1375/brim.10.1.45
Kliegel M, Eschen A, Thöne-Otto AI (2004) Planning and realization of complex intentions in traumatic brain injury and normal aging. Brain Cogn 56:43–54. https://doi.org/10.1016/j.bandc.2004.05.005
Knight RG, Titov N (2009) Use of virtual reality tasks to assess prospective memory: applicability and evidence. Brain Impair 10:3–13. https://doi.org/10.1375/brim.10.1.3
Kurtz MM, Baker E, Pearlson GD, Astur RS (2006) A virtual reality apartment as a measure of medication management skills in patients with schizophrenia: a pilot study. Schizophr Bull 33:1162–1170. https://doi.org/10.1093/schbul/sbl039
Kvavilashvili L, Ellis J (1996) Varieties of intention: some distinctions and classifications. In: Brandimonte M, Einstein GO, McDaniel MA (eds) Prospective memory: theory and applications. Lawrence Erlbaum Associates Inc, New Jersey, pp 23–51
Kvavilashvili L, Kornbrot DE, Mash V, Cockburn J, Milne A (2009) Differential effects of age on prospective and retrospective memory tasks in young, young-old, and old-old adults. Mem 17:180–196. https://doi.org/10.1080/09658210802194366
Man DW, Chan MK, Yip CC (2015a) Validation of the Cambridge prospective memory test (Hong Kong Chinese version) for people with stroke. Neuropsychol Rehabil 25:895–912. https://doi.org/10.1080/09602011.2014.997253
Man D, Yip C, Lee G, Fleming J, Shum D (2015b) Self-report prospective memory problems in people with stroke. Brain Inj 29:329–335. https://doi.org/10.3109/02699052.2014.974672
Man DWK, Ganesan B, Yip CCK, Lee COP, Tsang SYL, Yu PWP, Young JGE, Shum DHK (2018) Validation of the virtual-reality prospective memory test (Hong Kong Chinese version) for individuals with first-episode schizophrenia. Neuropsychol Rehabil 28:1197–1210. https://doi.org/10.1080/09602011.2016.1251949
Martin M, Kliegel M, McDaniel MA (2003) The involvement of executive functions in prospective memory performance of adults. Int J Psychol 38:195–206. https://doi.org/10.1080/00207590244000205
Maujean A, Shum D, McQueen R (2003) Effect of cognitive demand on prospective memory in individuals with traumatic brain injury. Brain Impair 4:135–145. https://doi.org/10.1375/brim.4.2.135.270
McDaniel MA, Einstein GO (2000) Strategic and automatic processes in prospective memory retrieval: a multiprocess framework. Appl Cogn Psychol 14:S127–S144. https://doi.org/10.1002/acp.775
Mitrovic A, Mathews M, Ohlsson S, Holland J, McKinlay A, Ogden S, Bracegirdle A, Dopping-Hepenstal S (2014) A virtual reality environment for prospective memory training. Paper presented at the meeting of User Modeling, Adaptation, and Personalization, Denmark
Mitrovic A, Mathews M, Ohlsson S, Holland J, McKinlay A (2016) Computer-based post-stroke rehabilitation of prospective memory. J Appl Res Mem Cogn 5:204–214. https://doi.org/10.1016/j.jarmac.2016.03.006
Morris R, Kotitsa M, Bramham J, Brooks B, Rose F (2002). Virtual reality investigation of strategy formation, rule breaking and prospective memory in patients with focal prefrontal neurosurgical lesions. Paper presented at the meeting of the International Conference on Disability, Virtual Reality and Associated Technologies, Hungary
Nasreddine ZS, Phillips NA, Bédirian V, Charbonneau S, Whitehead V, Collin I, Cummings JL, Chertkow H (2005) The Montreal cognitive assessment, MoCA: a brief screening tool for mild cognitive impairment. J Am Geriatr Soc 53:695–699. https://doi.org/10.1111/j.1532-5415.2005.53221.x
Nouri FM, Lincoln NB (1987) An extended activities of daily living scale for stroke patients. Clin Rehabil 1:301–305. https://doi.org/10.1177/026921558700100409
Otani H, Landau JD, Libkuman TM, Louis JPS, Kazen JK, Throne GW (1997) Prospective memory and divided attention. Memory 5:343–360. https://doi.org/10.1080/741941393
Rankin J (1957) Cerebral vascular accidents in patients over the age of 60: II. Prognosis. Scott Med J 2:200–215. https://doi.org/10.1177/003693305700200504
Rastle K, Harrington J, Coltheart M (2002) 358,534 nonwords: the ARC nonword database. Q J Exp Psychol Sect A 55:1339–1362. https://doi.org/10.1080/02724980244000099
Reitan RM (1992) Trail making test: manual for administration and scoring. Reitan Neuropsychology Laboratory Tucson, Arizona
Rendell PG, Henry JD (2009) A review of virtual week for prospective memory assessment: clinical implications. Brain Impair 10:14–22. https://doi.org/10.1375/brim.10.1.14
Rose FD, Brooks BM, Attree EA, Parslow DM, Leadbetter AG, McNeil JE, Jayawardena S, Greenwood E, Potter J (1999) A preliminary investigation into the use of virtual environments in memory retraining after vascular brain injury: indications for future strategy? Disabil Rehabil 21:548–554. https://doi.org/10.1080/096382899297206
Shum D, Fleming J, Neulinger K (2002) Prospective memory and traumatic brain injury: a review. Brain Impair 3:1–16
Smith RE (2003) The cost of remembering to remember in event-based prospective memory: investigating the capacity demands of delayed intention performance. J Exp Psychol Learn Mem Cogn 29:347–361. https://doi.org/10.1037/0278-7393.29.3.347
Sweeney S, Manly T, Kersel D, Morris R, Evans J (2010) The sensitivity of a virtual reality task to planning and prospective memory impairments: group differences and the efficacy of periodic alerts on performance. Neuropsychol Rehabil 20:239–263. https://doi.org/10.1080/09602010903080531
Wechsler D (2011) Test of premorbid functioning: UK version (TOPF UK). Pearson Inc, Bloomington Minnesota
Wilson B, Cockburn J, Baddeley AD (1985) The rivermead behavioural memory test. Thames Valley Test Company, Bury St Edmunds UK
Wilson B, Shiel A, Foley J, Emslie H, Groot Y, Hawkins K, Watson P (2005) Cambridge test of prospective memory (CAMPROMPT). Pearson Assessment, San Antonio
Wilson M (1988) MRC psycholinguistic database: machine-usable dictionary, version 2.00. Behav Res Methods Instrum Comput 20:6–10. https://doi.org/10.3758/BF03202594
Yip BCB, Man DWK (2013) Virtual reality-based prospective memory training program for people with acquired brain injury. Neurorehabil 32:103–115. https://doi.org/10.3233/NRE-30827
Acknowledgements
We thank Jayson Wockner-Pace and Cam Gibbs at VRSpace/GHD for their work on the VRPMST and Connor Reid at Griffith University for his work on the LDPMT. This work was supported by a Griffith University post-graduate scholarship.
Funding
This work was supported by a Griffith University Post-graduate Scholarship.
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by Dr. Christy Hogan. The first draft of the manuscript was written by Dr. Christy Hogan and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding authors
Ethics declarations
Ethical approval
Ethics approval was obtained from The Prince Charles Hospital Human Research and Ethics Committee, and the Griffith University Ethics Committee prior to commencing the study (Protocol Number: HREC/15/QPCH/282).
Consent to participate
Participants were treated strictly in accordance with the National Statement on Ethical Conduct in Research Involving Humans. Written informed consent was obtained by all participants at the start of the first session.
Consent for publication
Participants were informed that the results would be published in a way that their identity could not be revealed.
Data availability
Data set is available.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Hogan, C., Cornwell, P., Fleming, J. et al. Assessment of prospective memory after stroke utilizing virtual reality. Virtual Reality 27, 333–346 (2023). https://doi.org/10.1007/s10055-021-00576-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10055-021-00576-5