1 Introduction

1.1 Prospective memory

Cognitive impairments, including impairments in memory, are commonly reported after stroke (Andrews et al. 2014a). Much of the previous research investigating memory after stroke has focused on retrospective memory (RM; memory for past events and previously learned information) rather than prospective memory (PM; Andrews et al. 2014b). PM is the ability to remember to do something in the future and has been found to be impaired after stroke (Hogan et al. 2016). PM requires several interacting cognitive processes (Shum et al. 2002) with executive functions (EF) and RM playing important roles (Kant et al. 2014; Otani et al. 1997). The prefrontal lobes, in particular Brodmann Area 10, are often activated during PM tasks (Burgess et al. 2011); therefore, when this brain region and its surrounding areas are damaged due to stroke or injury, impairments in PM ability may occur. Impairments in PM can significantly impact ones’ independence and daily functioning (Man et al. 2015a) as PM failures can result in negative consequences for both individuals and those surrounding them. For example, if someone forgets to take their medication this may result in another stroke, or if an individual forgets to turn the stove off this may cause a house fire. Therefore, it is important to be able to correctly measure and identify PM impairments in order for clinicians to deliver appropriate rehabilitative techniques to help improve PM after brain injury.

Two main types of PM have been widely discussed in the literature: time- and event-based (Kvavilashvili and Ellis 1996). Time-based PM requires an individual to complete an action at a certain time or after a certain amount of time has elapsed (e.g., remembering to call your boss at 10:30 am or taking dinner out of the oven after 45 min), whereas event-based PM requires an individual to complete an action in response to a specific event that occurs in the individual’s environment (e.g., passing along a message to a family member when they return home). The multi-process framework proposes that in order for PM tasks to be completed correctly, an individual needs to use either a strategic monitoring approach to look for cues in the environment or rely on automatic spontaneous retrieval (McDaniel and Einstein 2000), with the monitoring approach dependent on the requirements of PM task requirements. It is believed that time-based PM would require greater cognitive resources for completion than event-based PM, because it depends on a strategic monitoring approach wherein an individual would continuously monitor the environment for a cue (i.e., regularly checking a clock for the time). Event-based tasks would require fewer cognitive resources for completion and usually rely on spontaneous retrieval (Einstein et al. 1995). When the cue/event that is embedded within the environment occurs (i.e., family member returning home) an individual would automatically remember that they needed to complete a PM task.

Many tools and paradigms have been designed to measure PM including experimental, naturalistic, clinical and Virtual Reality (VR), with each assessment type having their relative strengths and weaknesses. Ecological validity refers to the degree that a measure predicts daily functioning and the degree to which the cognitive demands required to complete the test resemble the cognitive demand used to complete a similar task in an individual’s everyday environment (Chaytor and Schmitter-Edgecombe 2003). In order for clinicians to reliably test one’s cognitive function, the tests used to measure these faculties must be reliable and valid, and reflect the individuals everyday performance.

While PM impairment has been reported after stroke (Hogan et al. 2016), results are mixed and dependent on the type of PM measure used or the PM type that is being measured. When PM after stroke is assessed using experimental measures (i.e., a dual-task laboratory paradigm that consists of both an ongoing and a PM task), impairments in time- but not event-based PM are often reported (Cheng et al. 2010; Kant et al. 2014). In the study conducted by Cheng et al. (2010), while completing an ongoing task, participants were required to tap the desk every 5 min for the time-based PM task, and tap the desk when they saw an animal word for the event-based PM task. Participants in the study conducted by Kant et al. (2014) were required to complete the Bourdon–Wiersma task (ongoing task) and say the word ‘regal’ aloud whenever the last configuration in a line contained three dots for the event-based PM task. For the time-based PM task, participants were instructed to insert a coin in a container after each minute has passed. While experimental PM tasks allow researchers to systematically manipulate parameters and provide good environmental control, they have been criticized for their lack of ecological validity. In addition, these tasks are usually tedious and repetitive for participants, particularly for individuals with brain injuries who may fatigue quickly. Finally, experimental measures are usually not standardized and do not have normative data, making comparisons and clinical judgments about impairment difficult.

Previous research that utilized single-item naturalistic measures generally did not find impairments in time (Kant et al. 2014) or event-based PM after stroke (Brooks et al. 2004; Kant et al. 2014; Kim et al. 2009). Kant et al. (2014) measured time-based PM by asking participants to remind the researcher to make a phone call after 30 min. Brooks et al. (2004), Kant et al. (2014), and Kim et al. (2009) utilized the Remember a Belonging Task from the Rivermead Behavioral Memory Test (RMBT; Wilson et al. 1985) to measure event-based PM. In this task participants have to give a personal item (i.e., a watch) to the researcher at the beginning of a testing session and are instructed to ask for the item back when the session concludes. One exception to these findings was reported by Brooks et al. (2004) who used an additional naturalistic measure of event-based PM wherein participants were required to ask for a written explanation of the study at the end of another task. On this task, individuals with stroke performed significantly poorer than healthy controls. While one would think that the best way to measure PM would be to get participants to complete real-life PM tasks, naturalistic tasks are often single-item measures which may not be robust enough to reliably measure PM alone (Kim et al. 2009). In addition, it is difficult for experimenters to control and manipulate the task parameters of naturalistic tasks. Furthermore, it may be hard to reliably measure performance for any tasks that are not completed in the presence of the experimenter as individuals may rely on other people or aids like a calendar alarm to remind them to complete the task.

A few psychological tests have been developed, standardized, and psychometrically validated to assess PM in clinical settings. When using the psychometrically validated Cambridge PM test (CAMPROMPT; Wilson et al. 2005) individuals with stroke were found to perform significantly poorer than controls on both time- and event-based PM (Barr 2011; Hogan et al. in press; Man et al. 2015b). The CAMPROMPT contains six different PM tasks (3 event- and 3 time-based) which are to be completed during or at the end of an ongoing paper and pen task. Research using another clinical measure, the Virtual Week which sees participants complete a number of time- and event-based PM tasks in response to cues provided by a game board (Rendell and Henry 2009), found that individuals with stroke showed impairments on time-based PM only (Kim et al. 2009). While clinical measures were developed with ecological validity in mind, the tasks that participants are required to perform in the majority of PM measures are still not typical of tasks that individuals are required to complete in their daily lives, resulting in the ecological validity being questioned (Canty et al. 2014).

While the results are mixed, the majority of the research suggests that PM exists after stroke, particularly for time-based PM. As previously outlined, time-based PM tasks require more cognitive resources for completion than event-based PM due to the type of monitoring required for task completion. Kant et al. (2014) found that individuals with stroke and controls who checked the time more frequently performed better on time-based PM, but individuals with stroke checked the time significantly less than controls, leading to poorer time-based PM performance. Therefore, deficits in monitoring may lead to PM impairment after stroke.

1.2 Virtual reality

Due to the limitations of current measures of neuropsychological function, the push for more ecologically valid measures, and advances in technology, VR has garnered considerable attention in recent years, offering possible solutions to these assessment problems (Knight and Titov 2009). VR comprises artificial computer-generated environments containing distinctive sensory properties and allows users to interact with the virtual environment in real-time (Mitrovic et al. 2016). VR can benefit neuropsychological assessment in many ways, for example, it can balance the demands of high ecological validity along with the requirements of sensitivity and specificity. VR platforms allow the embedding of naturalistic everyday PM tasks into environments that participants are familiar with, therefore, helping to overcome the practical and reliability issues that arise when assessing PM in real-life scenarios. Additionally, VR platforms provide the required conditions for standardized testing to take place, including a consistent environment, and potentially unlimited repetitions of a single assessment. At the same time, VR offers the flexibility to adjust specific aspects of a task, like complexity, difficulty, or response requirements for individuals, adjusting for possible impairments. Due to their novelty, VR environments are also believed to enhance test-takers’ motivation and interest (Sweeney et al. 2010; Canty et al. 2014) compared to tedious and repetitive laboratory tasks. Lastly, VR environments are particularly suited to assessing PM, because they can construct and facilitate a standardized procedure with ease but can also assess the dynamic coordination of multiple cognitive abilities simultaneously (Knight and Titov 2009).

Utilizing VR in the assessment of PM is an emerging field and has been used in multiple populations, including traumatic and acquired brain injury (TBI/ABI; Banville and Nolin 2012; Canty et al. 2014; Kinsella et al. 2009; Morris et al. 2002; Sweeney et al. 2010; Yip and Man 2013), stroke (Brooks et al. 2004; Mitrovic et al. 2016), schizophrenia (Kurtz et al. 2006; Man et al. 2018), aging (Debarnot et al. 2015), and healthy populations (Gonneaud et al. 2014). Many virtual environments have been developed, including shopping centers (Canty et al. 2014; Kinsella et al. 2009; Yip and Man 2013), apartments/houses (Banville and Nolin 2012; Brooks et al. 2004; Mitrovic et al. 2014; Rose et al. 1999; Sweeney et al. 2010), and cities/towns (Debarnot et al. 2015; Gonneaud et al. 2014). The literature has shown that VR platforms are suitable for use with clinical populations and can be adapted to suit the intrinsic needs of researchers and clinical populations.

1.3 Using VR to investigate PM after stroke

VR research investigating PM after stroke is limited with one study using a VR platform to assess PM (Brooks et al. 2004) and another using a VR platform in PM rehabilitation wherein participants could practice their newly learnt strategies to aid in prospective remembering. Brooks et al. (2004) compared PM performance between 25 individuals with stroke and 25 age-matched controls. The virtual environment consisted of a four-room bungalow wherein participants were instructed to help the owner pack to move house. In addition, they completed a series of event- (put ‘Fragile’ notes on items with glass components before moving them), activity- (close the door every time you leave the kitchen to keep the cat in), and time-based PM tasks (click on the red button in the hall next to the clock every 5 min to let the removalists into the house). It is important to note that participants did not control the virtual environment themselves. Rather the participant would tell the researchers what to do and the researcher would control the simulation. Therefore, it could be argued that the immersive feature of VR could be lost in the study. Brooks et al. found that the controls performed better than individuals with stroke on all PM tasks; however, significant differences were only found for event- and activity-based PM, whereas time-based PM was only marginally significantly different (p = 0.05). Moreover, the VR task was deemed to be more sensitive to PM impairments than the Remember a Belonging Task (Wilson et al. 1985).

It is important that standardized and ecologically valid PM measures, including VR measures, continue to be developed for use in both experimental and clinical settings. PM measures should aim to incorporate a variety of real-life tasks, naturalistic environments, platforms that engage participant’s attention, increase motivation, and feature user-friendly controls. These assessments should not only focus on the measurement of PM but also be able to manipulate and control certain factors to determine the underlying reasons or mechanisms of PM impairments.

1.4 Aims and hypotheses

This study aimed to design a VR task that measures both event- and time-based PM as well as time-monitoring (viz., Virtual Reality Prospective Memory Shopping Task; VRPMST) and use it to compare individuals with stroke and healthy controls. The study also aimed to compare the VRPMST to a psychometric (viz., CAMPROMPT), and experimental PM task (viz., Lexical Decision PM Task; LDPMT). Additionally, the VRPMST’s ecological validity in comparison to an experimental PM measure (viz., LDPMT) would be studied. Lastly, the study aimed to evaluate the convergent validity of the VRPMST by investigating the relationships between PM performance on the VRPMST PM performance and three standardized neuropsychological and two other PM measures.

H1: It was hypothesized that individuals with stroke would perform poorer than controls on both time- and event-based PM on all PM measures.

H2: Individuals with stroke would monitor the time significantly less than controls on both the LDPMT and VRPMST and that monitoring would significantly correlate with time-based PM performance.

H3: It was predicted that the VRPMST would be just as sensitive in measuring PM as the CAMPROMPT and LDPMT and have high ecological validity wherein both individuals with stroke and controls would rate the VRPMST more favorably than the LDPMT in terms of subjective user-friendliness.

H4: Lastly, it was hypothesized that the VRPMST would have strong convergent validity resulting in strong correlations between the VRPMST, neuropsychological measures, and the other PM measures (LDPMT and CAMPROMPT).

2 Method

2.1 Participants

Twelve individuals with stroke (6 females) were recruited through multiple stroke organizations and health facilities in South-East Queensland, Australia. Inclusion criteria included: between 18 and 85 years old, diagnosed cerebrovascular accident within the last five years, and living in the community at least one-month post-stroke. Exclusion criteria included: stroke located in cerebellum/brainstem, history of previous brain injury/neurological illness other than stroke, diagnosed dementia/neurodegenerative illness; significant psychiatric disorder; insufficient communicative ability (i.e., severe aphasia); significant visual/hearing/hand function impairment that would impact on participation ability; or current alcohol/substance abuse. On the Modified Rankin Scale (MRS; Rankin 1957) 3 individuals had no significant disability, 4 had slight disability, 4 moderate, and 1 moderately severe disability. On the Nottingham Extended Activities of Daily Living (NEADL; Nouri and Lincoln 1987) questionnaire, participants reported that they completed most activities of daily living independently or independently with difficulty (M = 53.92, SD = 13.98). Specific demographic stroke information is displayed in Table 1. Twelve control participants (9 females) aged between 18 and 85 years, with no history of neurological illness, brain injury, or stroke, were recruited through Griffith University (subject-pool, Learning Space, and research volunteer email) and various community groups (i.e., seniors’ groups). Exclusion criteria included: significant psychiatric illness, current alcohol/substance abuse, significant visual/hearing impairment, or insufficient communicative ability. The two groups did not differ on any demographic variable (Table 1).

Table 1 Demographic information for individuals with stroke and neurologically healthy controls
Full size table

2.2 Measures

2.2.1 VRPMST

The VRPMST, developed for the current study, was adapted from the VR framework (shopping center) used by Canty et al. (2015). The VRPMST features a dual-task paradigm (ongoing and event- and time-based PM tasks) run on a laptop, with an iPad used to wirelessly control the simulation (Fig. 1). The virtual environment was developed and adapted by a team of Computer Graphic Artists and Programmers at VRspace Pty Ltd. The digital assets (i.e., characters and shopping center) were created using Auto-desk 3D Studio Max and the functionality of the application was developed using Unity3D game engine. Participants completed the task on a Dell M6700 note-book computer. Hardware requirements are Windows 7, 3.0 GHz CPU, a dedicated Graphics Card/an adaptor with at least 2 Mb texture memory and 4 Gb RAM. In addition, a wireless router connects the iPad which is used to control the simulation from an app built by the programmers to the laptop via Wi-Fi. The laptop is placed in front of the participant and adjusted so that they can comfortably view the screen.

Fig. 1
figure 1

Map of the virtual shopping environment. The red dot indicates where the participant is located and the green arrows indicate shop entrances. The control screen is displayed on an iPad used to control the VR simulation

Full size image

A practice trial (5 min) where participants can become familiar with the controls (locate and view the map, list, and watch) and complete an example ongoing task (i.e., enter a shop and complete a task), as well as receive confirmation of instructions are completed before starting the assessment, ensuring that participants understand the task requirements and are familiar with the navigation and response controls. The ongoing task (12 errands) is completed by moving around the virtual shopping center and entering stores in a set order. The tasks are designed to mimic everyday activities completed at a shopping center (e.g., buy a birthday card from the newsagent). To facilitate immersion, animated advertisement boards and background chatter are built into the ongoing task. For time-based PM, participants are told that their doctor has instructed them to keep track of their heart rate; therefore, they need to check their heart rate (press H on the iPad) every 3 min. Participants can check their virtual watch (elapsed time appears for three seconds; Fig. 2) by pressing the TIME button on the iPad. Time-based PM is scored as deviation in response to the target time (i.e., 3 points =  ≤ 15 s after target time, 2 points =  > 15 but ≤ 30 s, 1 point =  > 30 but ≤ 60 s, 0 points =  > 60 s after specified time or no response at all) with similar methods used in previous PM research (Canty et al. 2014; Einstein et al. 1995). Monitoring is measured as the overall number of clock checks.

Fig. 2
figure 2

VRPMST time-based PM task with both the iPad and computer view. Participants can check the elapsed time on their virtual watch by pressing TIME and check their heart rate by pressing H. Red circles indicate a participants’ correct response (not shown during the task)

Full size image

The VRPMST features two types of event-based PM tasks. Firstly, participants are told that they are keeping track of food expenditure. Therefore, after purchasing food items (4 items), they are required to collect a receipt (by pressing R on the iPad; Fig. 3). Secondly, participants are informed that they lost their glasses on their previous shopping trip. Therefore, they need to ask the center’s security guard every time they see one (4 times) whether they have found their glasses (press S on the iPad; Fig. 4). Event-based PM is scored out of 8, with higher scores indicating better performance.

Fig. 3
figure 3

VRPMST receipt event-based PM task with both iPad and computer view. Participants can collect a receipt by pressing R after buying an item of food. Red circles indicate a participants’ correct response (not shown during the task)

Full size image
Fig. 4
figure 4

VRPMST security guard event-based PM task with both iPad and computer view. Participants can ask about their lost glasses by pressing S after seeing the security guard. Red circles indicate a participants’ correct response (not shown during the task)

Full size image

2.2.2 CAMPROMPT

The CAMPROMPT (Wilson et al. 2005) is a standardized psychometric PM measure commonly used in experimental and clinical settings and was used in the current study to obtain a baseline measure of PM. It features three event- and three time-based PM tasks to be completed throughout or at the end of the ongoing paper puzzle task. The CAMPROMPT has a parallel form, excellent inter-rater reliability (r = 0.99), test–retest reliability (Kendall's tau-b = 0.64) and has been used in previous stroke research (Barr 2011; Man et al. 2015b).

2.2.3 LDPMT

Developed for the current study and adapted from Canty et al. (2014) and Maujean et al. (2003), the LDPMT features a dual-task paradigm run on an iPad. Event-based PM is assessed first followed by a break period to avoid fatigue, then time-based PM, with practice trials and confirmation of instructions before each section. This ensures participants' understanding of the task requirements and familiarity with controls. Section one features an ongoing Lexical Decision Task (LDT) where participants are to determine whether the text on the screen is an English word or a non-word. Participants are shown a series of 55 English words sourced from the MRC Psycholinguistic Database (Wilson 1988), and 44 legal non-words that follow English language rules sourced from the ARC Nonword Database (Rastle et al. 2002), and 6 event-based PM animal words in a pre-set random order. If the text on the screen is an English word (e.g., GIRL) participants are to press the left button on the iPad. If the text is a non-word (e.g., YIMS), they are to press the middle button and if the text is an animal word (e.g., FROG) they are to press the button on the right. Each item is four letters long, presented in capital letters in a clear legible font and displayed for 3 s then disappears; however, participants can take as long as needed to respond. After responding, a plus sign (+) is displayed center-screen for 1 s, followed by the next trial. The ongoing LDT is scored as percentage correct (number of correct responses divided by 99 trials, multiplied by 100), and event-based PM is scored out of six, with higher scores indicating better PM performance.

For time-based PM, participants complete another LDT (i.e., determine whether the text on the screen is a word or a non-word, but, unlike the event-based task, no animal words are included) wherein they are required to press the button on the right every 60 s. Six time-based PM trials are placed throughout the task (i.e., 1 min, 2 min…etc.) and the assessment automatically ends at 6 min and 30 s. To check the elapsed time, participants can press a gray bar at the top of the screen as often as they like. The time is presented for 2 s then disappears. LDT percentage correct score is calculated (number of correct responses divided by total trials completed, multiplied by 100), and time-based PM is scored out of 18, with higher scores indicating better performance. Time-based PM is scored as deviation in response to the target time (i.e., 3 points =  ≤ 5 s after target time, 2 points =  > 5 but ≤ 10 s, 1 point =  > 10 but ≤ 15 s, 0 points =  > 15 s after specified time or no response at all). While LDT tasks have not been previously used in PM assessment post-stroke, they have been widely used in the experimental PM literature (Einstein et al. 1995; Ellis and Milne 1996) and in clinical populations like TBI (Canty et al. 2014; Maujean et al. 2003); therefore, they are a good criterion to validate the VRPMST against.

2.2.4 User-friendliness scale (UFS)

The UFS-LDPMT (eight items) and UFS-VRSPMT (9 items) were adapted from Canty et al. (2014) to assess participants’ subjective testing experience of the LDPMT and VRPMST using a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree). The UFS assesses to what degree the measures reflected everyday activities, was interesting, clear and easy to learn, task difficulty, and if they would recommend the task.

2.2.5 Neuropsychological measures

The participants completed a number of neuropsychological tests commonly used in clinical and experimental settings to assess executive function (Trail Making Task; TMT; EF component = Part B minus Part A; Reitan 1992), verbal learning and memory (Hopkins Verbal Learning Test-Revised; HVLT-R; Benedict et al. 1998), and global cognitive function (Montreal Cognitive Assessment Scale; MoCA; Nasreddine et al. 2005) that covers attention and concentration, EF, memory, language, visuoconstructional skills, conceptual thinking, calculations, and orientation. These measures were selected to provide evidence of convergent validity as the processes they assess are related to PM (Canty et al. 2014; Fleming et al. 2008; Kant et al. 2014; Martin et al. 2003) and have previously been used in stroke populations (Andrews et al. 2014b; Cumming et al. 2011; Kant et al. 2014).

2.3 Procedure

Prior to study commencement, ethical clearance was obtained from the relevant hospital and university ethics committees and written informed consent was obtained from all participants before the first session. An assessment battery was conducted over two sessions, each taking approximately 1.5–2 h with breaks if required to avoid fatigue. Assessment sessions were held in the participants’ homes or in a quiet room at the university. As testing took place in an office or in individuals’ homes, the testing conditions were not the same for each participant; however, it was ensured that there was adequate lighting and that each testing session was free from excess noise/distractions. After completion of both sessions, participants were compensated $20 or 2 credit points if recruited through the university’s subject-pool.

2.4 Data analysis

The data were analyzed using IBM SPSS Statistics for Windows (Version 25; IBM Corporation 2017). The data were screened for accuracy, missing values, outliers, and normality. Descriptive statistics were obtained for basic demographic data and are outlined in the participants section. Comparisons between the stroke and control groups on the VRPMST, CAMPROMPT, and LDPMT were made using independent-samples one-tailed t tests. Effect sizes were determined using Cohen’s d: small = 0.2, medium = 0.5, and large = 0.8 (Hu 2010). Independent samples t tests were used to analyze the results between groups on the UFS-LDPMT, and UFS-VRPMST. Paired-samples t tests were used to compare the LDPMT and VRPMST on user-friendliness. Pearson’s correlational analyses were used to examine the relationships between the VRPMST and other PM measures (CAMPROMPT and LDPMT) and neuropsychological measures (TMT, HVLT-R, and MoCA). Effect sizes for correlations were interpreted as: small = 0.1, moderate = 0.3, and strong = 0.5 (Cohen 1998).

3 Results

Due to technical errors, one control LDPMT and one stroke VRPMST data point were missing. One individual with stroke had missing data for time-based LDPMT due to fatigue-related noncompletion. The VRPMST ongoing (ceiling effect for controls), LDPMT event-based (stroke), UFS-VRPMST item 5 (ongoing task; ceiling effect for controls), and UFS-VRPMST item 9 (would recommend to others; stroke) variables were significantly skewed. Transformations were applied; however this did not change the results; therefore, the raw data is reported. One outlier (control) for item 4 (task easy to learn) of the UFS-VRPMST, one outlier (control) on the TMT, and two outliers (controls) on LDPMT monitoring were removed from the dataset. Two outliers (control and stroke) were found on item 6 (found event-based PM difficult) on the UFS-LDPMT; however they did not impact on the results; therefore the raw data is reported. Table 2 summarizes the means, standard deviations, and t tests between groups on all PM and neuropsychological measures.

Table 2 Means and standard deviations for each group on all PM and neuropsychological measures
Full size table

3.1 PM performance between groups

3.1.1 VRPMST

Both groups scored similarly on the ongoing task, indicating that they did not find the task too difficult. Controls performed significantly better on time-based PM compared to individuals with stroke (medium effect size). Additionally, controls monitored the time significantly more than individuals with stroke (large effect size). Controls scored higher than individuals with stroke on event-based PM; however, no significant difference was found. VRPMST monitoring significantly correlated with VRPMST time-based PM for both groups (stroke: r = 0.86, p < 0.001; control: r = 0.70, p = 0.012).

3.1.2 CAMPROMPT

Individuals with stroke performed significantly worse on the CAMPROMPT compared to controls on both event- and time-based PM (large effect sizes).

3.1.3 LDPMT

Both groups scored similarly on the ongoing tasks of the LDPMT (p > 0.05 for both event- and time-based LDTs), scoring above 89% correct. The stroke group performed poorer than controls on event-based PM; however the difference was not significant (medium effect size). Controls performed significantly better than individuals with stroke on time-based PM (large effect size) and were found to monitor significantly more than individuals with stroke (large effect size). Monitoring on the LDPMT was found to significantly positively correlate with time-based PM for individuals with stroke (r = 0.66, p = 0.029; strong) but not significantly correlate for controls on time-based PM (r = -0.25, p = 0.52; moderate).

3.2 User friendliness

User-friendliness was examined for the LDPMT and VRPMST (Table 3). Ease of understanding on the LDPMT was significantly different between groups with controls finding it easier to understand; however the mean scores for both groups represented agree. Ease of learning on the LDPMT was significantly different between the groups, with stroke reporting a neutral to agree response compared to the controls agree to strongly agree response; therefore the results are still favorable for the LDPMT. Individuals with stroke reported a neutral to agree response when recommending the LDPMT, significantly different from the controls response of agree to strongly agree. Controls reported that the VRPMST included everyday activities and was easy to understand significantly more than individuals with stroke; however both groups mean responses ranged between agree and strongly agree for both items. To examine the differences between the subjective ratings of the LDPMT and VRPMST, paired-samples t tests were conducted (Table 4). Both groups rated the VRPMST to contain significantly more activities that were similar to everyday life compared to the LDPMT.

Table 3 User friendliness scale for the LDPMT and VRPMST between groups
Full size table
Table 4 Subjective Experience Rating of the LDPMT Compared to the VRPMST
Full size table

3.3 Correlational analyses

Individuals with stroke performed significantly worse than controls on the TMT (large effect size) and both HVLT-R scores (large effect sizes; Table 2). No significant differences were found between the groups on the MoCA. Correlational analyses were conducted to investigate the relationships between performance on the VRPMST, other PM measures, and neuropsychological measures (see Table 5). Time- and event-based VRPMST scores were significantly strongly correlated with both time- and event-based PM on the CAMPROMPT for the stroke sample. For controls, only event-based VRPMST significantly strongly correlated with both time- and event-based PM on the CAMPROMPT. Significant strong correlations were found between VRPMST time-based PM and LDPMT event- and time-based PM for the stroke sample but not for the control group. For individuals with stroke event-based PM on the VRPMST was significantly strongly correlated with the TMT, both HVLT-R scores, and MoCA, while time-based PM was significantly strongly correlated to the TMT, HVLT-R total recall score, and the MoCA. For controls VRPMST event- and time-based PM was significantly strongly correlated to TMT; however only the HVLT-R percent retained score was significantly correlated with time-based PM.

Table 5 Correlational analyses between the VRPMST and other PM and neuropsychological measures
Full size table

4 Discussion

This study aimed to design a PM measure using VR to assess individuals with stroke and healthy controls and compare the results to a standardized and an experimental measure of PM. Hypothesis one, that individuals with stroke would perform poorer than controls on time- and event-based PM on all PM measures, was partly supported. Consistent with previous research (Barr 2011; Hogan et al. 2020; Man et al. 2015b) individuals with stroke performed significantly poorer than controls on both event- and time-based PM on the CAMPROMPT. On the VRPMST and LDPMT individuals with stroke performed significantly poorer than controls on time-based PM. While individuals with stroke performed worse than controls on event-based PM on both the LDPMT and VRPMST, no significant differences were found between the groups.

The results of the current study suggest that PM impairment does exist after stroke, particularly for time-based PM, which is in line with previous research (Hogan et al. 2016). While previous literature has not utilized LDTs in the assessment of PM after stroke, similar studies using dual-task paradigms have been conducted. The current findings are consistent with those of Cheng et al. (2010) and Kant et al. (2014) who found that individuals with stroke performed significantly poorer on time- but not event-based PM compared to controls when assessed with experimental PM measures. These findings are most likely due to the difference in measure type and PM task type, as event-based PM is believed to require fewer cognitive resources for completion than time-based PM (Einstein et al. 1995). This is because event-based PM tasks have more environmental cues than time-based tasks, as such they require less self-initiation. The general consensus is that time-based tasks demand more resources than event-based; however, it is possible that event-based PM tasks could be equal to or require more resources than time-based tasks dependent on the task itself and the availability of PM cues within the environment. However, for the current study, the subjective ratings by both group of participants suggest that time-based tasks were more difficult than the event-based PM tasks.

Only one previous study has used VR to assess PM in stroke populations (Brooks et al. 2004). While the previous research found significant differences between individuals with stroke and controls in event-based PM and a marginal difference (p = 0.05) in time-based PM, the current study found differences in time-based PM but not event-based PM. One possible explanation for inconsistent findings may be the difference in cue saliency or ability to monitor the time between the two different PM measures. The TIME button was clearly displayed on the iPad in the current study. However, in the study conducted by Brooks et al., to check the time, participants had to ask the experimenter to enter the hall and press a red button next to a digital clock. Therefore, in the current study that participants may have been prompted to check the clock more often than in the Brooks study as the cue was in plain sight at all times while completing the test. It could be suggested that both individuals with stroke and controls in the Brooks study found the time-based PM task harder due to the clock not being in direct view at all times. The hypothesis that individuals with stroke would monitor the time significantly less than controls on both the LDPMT and VRPMST was supported. Additionally, time monitoring was found to significantly correlate with time-based PM on both the LDPMT and VRPMST for individuals with stroke, with greater monitoring resulting in higher PM performance. These results suggest that PM impairment after stroke may be due to deficits in strategic monitoring ability, which would negatively impact time-based PM, and possibly event-based PM tasks that lack salient cues.

The current study also aimed to corroborate the ecological validity of the VRPMST in comparison to a conventional experimental PM measure (LDPMT). The hypothesis that both individuals with stroke and controls would rate the user-friendliness of the VRPMST more favorably than the LDPMT was supported. Overall, both measures were rated favorably by both groups. Participants agreed that the tasks were interesting, instructions were clear and easy to understand, tasks were easy to learn, and that they would recommend the tasks to other people. Individuals with stroke reported that they would recommend the VRPMST just as much as the LDPMT. This finding suggests that VR is suitable to use in stroke populations and positively enhanced the participants’ testing experience. The most important finding was that both groups deemed the VRPMST to contain more tasks that were similar to everyday life than the LDPMT, suggesting higher ecological validity. The VRPMST was also found to be sensitive to the assessment of PM, particularly time-based PM, as the results were similar to those found on the CAMPROMPT and LDPMT. As the VR measure was rated favorably and similar to the experimental PM measure, and individuals with stroke were able to control and engage with the virtual environment, VR measures should continue to be developed and used in stroke populations.

On both the LDPMT and VRPMST, the groups did not find the ongoing task difficult, evidenced by UFS self-reports and high scores, accompanied by no significant difference between groups when questioned about task difficulty. These results suggest that the ongoing task was unlikely to impact on PM performance as they did not require high levels of cognitive demand. On both the LDPMT and VRPMST, both groups reported the time-based PM tasks to be more difficult than the ongoing and event-based PM tasks. Einstein et al. (1995) suggested that time-based PM is harder to complete most likely due to participants needing to use a strategic monitoring approach to continually monitor the time, further suggesting that time-based PM may be impaired due to monitoring deficits after stroke. Overall both groups reported that they did not find any of the PM tasks overly difficult (strongly disagree to neutral).

The last objective of the study was to evaluate the convergent validity of the VRPMST by exploring its relationships with neuropsychological and other PM measures. The study found evidence of convergent validity for the VRPMST and partially supported the final hypothesis that stated that the VRPMST would have strong correlations with neuropsychological measures and other PM measures. For controls, only time-based PM was significantly correlated with a measure of RM while both event- and time-based PM was significantly correlated with EF. For individuals with stroke both event and time-based PM on the VRPMST was significantly correlated with a measure of RM, EF, and global cognitive function. This is consistent with previous literature that suggests that RM and EF are related to experimental time-based PM and RM to event-based PM after stroke (Kant et al. 2014). Within TBI samples, previous research has found EF to be predictive of event-based PM (VRST and CAMPROMPT; Canty et al. 2014; Fleming et al. 2008, respectively) and EF and RM to be related to total PM performance (combination of event- and time-based PM measured by VRST; Canty et al. 2014). The current findings further support the notion that executive processes and intact RM are required for PM completion (Fish et al. 2007; Kliegel et al. 2004; Kvavilashvili et al. 2009). Further preliminary evidence of the convergent validity was suggested by strong significant correlations observed between the event- and time-based PM scores on the VRPMST and LDPMT for individuals with stroke. LDT tasks have been used as a benchmark tool in the experimental PM literature (Einstein and McDaniel 1990; Ellis and Milne 1996); therefore finding relationships between the VRPMST and the LDPMT provides preliminary support for the convergent validity of the VRPMST. Further evidence of the convergent validity of the VRPMST was suggested by the strong significant correlations to the CAMPROMPT, a psychometric measure of PM, for individuals with stroke.

Overall impairments in both time- and event-based PM were found for individuals with stroke compared to controls, suggesting impairments in PM after stroke. The current findings add to the little research that has been conducted focusing on PM after stroke, highlighting potential impairments and that more research needs to be conducted to determine whether PM impairment exists after stroke and what are the possible reasons for impairment. This will enable clinicians and researchers to assist in the rehabilitation of PM after stroke. The small sample size is a limitation; therefore, the current study may be underpowered to find significant results. Future research should aim to recruit larger numbers of individuals with stroke and age and education matched controls in order to complete more sophisticated statistical analyses which may provide more concrete conclusions. A larger sample size would also allow researchers to control for certain confounding or predictive factors like age, severity of stroke, or location of brain injury. In addition, both the LDPMT and VRPMST experienced some minor technical errors resulting in some missing data. While the current research has its limitations, it has helped to clarify the findings of previous research in that PM impairment exists after stroke, particularly time-based PM (Barr 2011; Cheng et al. 2010; Kant et al. 2014; Man et al. 2015b). Impairments may be due to deficits in strategic monitoring ability. While monitoring was easy to measure in the current study for time-based PM (i.e., total number of clock checks) future research should aim to examine monitoring strategies in both time- and event-based PM, possibly through the use of eye-tracking technology. The findings also suggest that VR can be used to assess PM in both clinical and healthy populations and that the VRPMST is sensitive in measuring PM. The VRPMST was also found to be an ecologically valid tool when compared to the LDPMT and have good convergent validity when assessed against neuropsychological tests and the CAMPROMPT, respectively. Given the benefits of using VR to assess PM and the favorable ratings provided by both groups in the current study, it is suggested that future research continues to develop and improve the paradigms to accurately and effectively measure PM after stroke and in other clinical populations.