Introduction

Neurosurgery has always been at the forefront of incorporating technological advances to better surgical management, with the incorporation of augmented reality (AR) and virtual reality (VR) in recent years [5, 59, 82]. 3-D AR and VR assist with complex neurosurgical procedures in training neurosurgeons, planning surgeries, and helping with patient recovery following the surgery, respectively through projection of virtual content in the real world and generating a 3-D environment where users can fully immerse themselves [5, 82]. These two modalities have been used in multiple neurosurgical sub-specialties and have aided preoperative training and planning, surgical decision-making, intraoperative workflow, risk minimisation, postoperative clinical assessments, and rehabilitation [12, 17, 20, 30, 39,40,41, 45, 50, 57, 60, 63, 65, 66, 79, 89].

Computer-generated realistic images, sounds, and other sensations in VR immerse users in artificial and synthetic environments that simulate a desired situation [17, 20, 39, 40, 65]. Given that VR users have the ability to explore the artificial environment, move within it, and interact with other users [16], a simulation could be devised that enables neurosurgeons to train and plan for conducting an awake craniotomy (AC). Intraoperative brain mapping using direct electrical stimulation of cortical and subcortical regions can be performed in AC to reduce the risk of permanent loss of neurological functions in eloquent areas, such as areas associated with language, motor, or vision by real-time monitoring of neurological functions with the patient participating in different tasks [26, 29, 42, 48, 53, 54, 87, 88]. In language mapping, for example, the neurosurgeon uses direct electrical stimulation to stimulate cortical and subcortical areas, while the patient is asked in real time to name images presented to them. Perturbation or arrest of language fluency upon stimulation of certain areas marks identification of language networks and subnetworks [46]. Similarly, mapping of motor areas can be accomplished by asking patients to perform certain motor tasks [71, 76, 85]. However, other important neurological functions in social interactions, such as facial expression and eye gaze, are not mapped, mainly due to bedside restrictions in the operating room to incorporate neurophysiological functions [9]. This limitation is resolved via VR use.

Incorporating relevant information into the surgical field has been proposed for several years [1, 37]. Since its introduction to neurosurgery, in 1986, AR has become more prevalent in clinical developments such as frameless neuronavigation to help guide interventions [30, 31, 74]. Despite significant technological advances over the past few decades, current navigation approaches require manipulating surgical instruments in the surgical field while mentally reconstituting 2-D information specific to patients, such as computed tomography (CT) or magnetic resonance imaging (MRI), to 3-D anatomy in the surgical field [30]. Conventional neuronavigation tools with a screen in the operating room require exchanging surgical instruments, holding a navigation pointer in one hand, and alternating the viewing between the surgical field and the screen, therefore causing attention shift [6, 7, 28].

VR can be used in the simulation for the training of surgeons, improving understanding of complex anatomical structures, and expanding clinical assessments [39, 40, 46, 65, 82]. This is a noted advantage since it creates a risk-free setting for the clinical training of neurosurgeons across different levels to reduce the learning curve [17, 82].

A difference between the two modalities is that in an AR setup, computer-generated 2-D and 3-D images are superimposed into the vision of the real world [39, 62], whereas there is no real-world input in VR, and the user is fully immersed in a computer-generated environment [39,40,41]. While AR and VR have been more frequently used in other neurosurgical applications, their usage in AC remains relatively limited. In this systematic review, our primary purpose is to investigate the applications of AR and VR in AC, which can open discussions and facilitate their usage in AC. To the best of our knowledge, this paper is the first systematic review that accomplishes a synthesis of the published peer-reviewed literature on AR and VR in AC.

Methods

Search strategy

We conducted our systematic review based on the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines [68] to identify published literature investigating AR and VR in AC. We conducted our electronic searches using PubMed, Scopus, and Web of Sciences from inception to May 20th, 2022, for relevant articles. The following Boolean terms were used for the search: (“augmented reality” OR “virtual reality” OR “extended reality” OR “mixed reality” OR “virtual simulation”) AND (“awake craniotomy” OR “awake brain surgery” OR “awake neurosurgery” OR “awake brain mapping” OR “awake tumour resection”) in different combinations. Details of search terms for each database are shown in Supplementary Table 1. After duplicates were removed, titles and abstracts of search results were screened, and non-related articles were excluded. Full-text articles were assessed for eligibility. Three authors (M.M., M.S.M., and S.A.) screened for relevant articles of the reference lists of selected articles to ensure no additional relevant articles were excluded.

Inclusion and exclusion criteria

We considered articles eligible for our systematic review if they met the following conditions: (1) original articles, (2) published in English only, (3) used at least one aspect of AR and/or VR in AC, and (4) exclusively involving human subjects. The exclusion criterion was defined as (1) studies which investigated neurosurgical interventions other than AC, for example, minimally invasive procedures such as burr holes in deep brain stimulation [32, 56].

Data extraction

Extracted data are presented in Tables 1, 2, 3, and 4 and Supplementary Table 2. All calculations were done on Microsoft Excel (version 2016; Microsoft, Redmond, WA, USA).

Table 1 An overview of studies’ characteristics and patients’ demographics
Full size table
Table 2 An overview of patients’ demographics
Full size table
Table 3 A summary of AR/VR details from studies reviewed
Full size table
Table 4 Major outcomes and complications from studies reviewed
Full size table

Results

Literature search results overview

Details of our search results are shown in Fig. 1. From six studies included, four papers were from France (66.67%) [2, 9, 18, 61], and two were from Germany (33.33%) [35, 75] (Supplementary Table 2).

Fig. 1
figure 1

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart demonstrating the search, screen, inclusion, and exclusion process for the current study. AC, awake craniotomy; AR, augmented reality; VR, virtual reality

Full size image

Study characteristics and patients’ demographics

The total sample size was 118, with the largest sample size of 55 patients reported by Roethe et al. [75], and the smallest cohort consisted of three patients [2] (Table 1). The youngest mean cohort age was noted at 45 [18], while the oldest mean age was 57.7 years [61]. The youngest patient in any cohort was 14, and the oldest was 81, both were in the study by Mazerand et al. [61]. With the exception of the 14-year-old, all studies focused on adults. Five studies specified the sex of participants [9, 18, 35, 61, 75]; there were 64 males (54.24%) and 49 females (41.53%).

The majority of pathologies operated on by AC were tumours (n = 108, 91.53%), followed by vascular lesions (n = 8, 6.78%) (Table 2). The pathology was not specified in two cases (1.69%). Five studies (83.3%) specified the lesion hemisphere and location [2, 9, 18, 61, 75]. Seventy-three lesions (61.86%) were located in the left hemisphere, and the most common lesion location was further specified as the frontal lobe (n = 38, 32.20%). Three studies reported preoperative symptoms in their patients, with seizure being the most common symptom recorded in 26 patients (22.03%) [9, 18, 61]. Finally, language mapping of eloquent areas was carried out in all studies. Additional mapping of motor [2, 9, 61, 75], social cognition [2, 9], and vision [61] were reported in other studies.

Application of AR and VR in AC

Applications of AR and VR were specified with four papers studying VR [2, 9, 18, 61] and the other two investigating AR [35, 75] (Table 3). Among VR studies, two papers focused on the usage of VR for patients [9, 61], and the other two used VR for patients and staff (neuropsychologists) [2, 18]. The main uses for VR were assessing the visual field and mapping optic radiation [61] as well as mapping language and social cognitive function, including eye gazing and facial expression [2, 9, 18]. In contrast, AR was utilised for training white matter dissection [35], navigation, and visualisation of brain lesions [75]. Both AR studies involved staff, including surgeons [35, 75].

Two studies reported the mean surgery time, which were recorded as 4 h and 12 min (AC phase, 2 h and 12 min) [18], and 4 h and 23 min (AC phase, 2 h and 20 min) [9]; however, they did not specify whether AR/VR resulted in any changes in surgery duration compared to conventional methods.

Fifteen patients in three studies had previous experience using VR [2, 9, 18]. Appropriate training and instructions were provided for participants to use AR and VR in four studies [2, 9, 18, 75]. Four studies compared AR and VR to different conventional methods present [9, 18, 61, 75]. While all VR setups compared to conventional methods performed equal or better [9, 18, 61], some functions were scored equal or worse in AR compared to other conventional neuronavigation techniques [75].

Virtual reality in awake craniotomy

While brain mapping during AC is mainly focused on language and motor areas [70, 95], VR may offer novel opportunities for mapping other complex cognitive functions, such as vision and social cognition [46]. Mazerand et al. [70] reported the first study on using VR headsets in identifying functional brain areas and functional cognitive and visual assessments during AC (Table 4). They developed a headset function similar to the Esterman test to detect homonymous and congruous visual field impairments to assess binocular visual field defects intraoperatively in AC [15]. In this setting, luminous stimuli are provided on the screen of a VR headset by selecting the points on a computer screen among those suggested by the modified Esterman test awaiting the patient’s response while direct cortical and/or subcortical stimulation was performed by the neurosurgeon in this setup. The patient communicates orally whether s/he has seen the luminous stimuli. Comparing conventional technique to VR on ten patients showed that nine patients (90%) were well-classified using the VR headset.

Bernard et al. [2] used VR to enable patients to interact with a neuropsychologist avatar for mapping their social cognition during AC. Patients observed the neuropsychologist’s avatar to comment on and reproduce gestural signs, such as OK, thumbs up, and clap. Bernand and colleagues concluded their study with the finding that VR could imitate complex social interactions in patients undergoing AC. The advantage of using VR to map social cognition is a fast and definite response by patients compared to other mapping approaches, such as story movies, comic strips, or interactive games, which are not compatible with a fast duration of direct electrical stimulation within a few seconds [2]. Furthermore, patient becomes an active participant in the environment rather than a mere passive observer during brain mapping procedures.

DO-80 language denomination test, where patients are asked to name 80 objects presented to them, has been extensively used for mapping language areas [11, 91], and has been incorporated into VR [9]. Delion et al. [18] duplicated the DO-80 picture naming task in the VR headset in two versions to avoid interfering with routine language mapping and AC procedures. The first version used the same images as the classic DO-80 in a virtual 2-D open space, and the other one used the 3-D version of the same objects rotating in a virtual open space. The same number of eloquent areas was identified, regardless of whether classical or VR DO-80 tests were used. However, using a VR headset successfully helped to clarify the absence of eloquent areas in three patients, which was not clear using the classical DO-80 test due to hesitation or delays in denominations. The authors speculate that VR headsets can assist patients with a better 3-D visualisation and help them focus without the distractions of the operating theatre. Furthermore, Delion and colleagues used a low-cost, high-quality, and customisable VR headset, which heated up after lengthy usage, with the phone battery acting as the rate-limiting factor [18]. While these issues did not affect their setup, such weaknesses should be considered and addressed in future investigations. In addition, placing the VR headset may interfere with the Mayfield skull clamp, which can be resolved to some extent by positioning the headset before the head holder and careful marking of incision lines.

Casanova et al. [9] used the same items as in the DO-80 language test but in a VR 3-D, similar to the setup by Delion et al. [18]. The same eloquent language areas could be identified using both conventional or VR-based language mapping, although confirmation about the absence of an eloquent area was achieved in one patient using a VR headset and not by the conventional DO-80 test. They extended their VR setup further by designing a new test for mapping eye gazing and facial expression during AC using avatars. Various types of eye movement data, such as pupil size and position as well as gaze direction, were collected using their VR headset. In their novel VR social cognition task, five different avatars were presented to patients in front of a landscape background, and each patient was asked to look at the avatar that was staring at them. Once the eye contact was established, the avatar randomly expressed a facial gesture (i.e. joy, anger, surprise) within a second after the visual contact, and patients were asked to correctly identify and describe the avatar’s emotion. This VR setup enabled establishing whether the patient had difficulties exploring the environment and locating the avatar or could not identify the facial expression after direct electrical stimulation. Eloquent areas were defined if direct electrical stimulation resulted in difficulties in exploring the VR environment, locating the avatar making eye contact, recognising the facial emotion, causing a delay, or lack of response from the patient for three times. The mean time to identify the eye contact and recognise and communicate the facial expression was 2.3 s (range, 2.0–2.5 s) and 3.2 s (range, 2.6–4.4 s), respectively, without direct electrical stimulation [18]. Particularly, the AC patient was expected to answer the questions within the time frame of each direct electrical stimulation, which lasted only 5 s.

VR sickness is accompanied by experiences such as nausea and headache [83], and may be a limiting factor for VR usage. While three studies reported no VR sickness in their participants, it is important to identify patients with risk factors to prevent adverse effects [2, 9, 18].

Augmented reality in awake craniotomy

AR can be used for neurosurgical training. Surgical treatment of eloquent gliomas requires sufficient knowledge of the white matter to preserve these structures during tumour resection [21, 38, 80]. Current training programmes use a combination of cadaver dissection, white matter tractography on MRI, and direct electrical stimulation during AC [10, 24, 25, 33]. Ille et al. [35] used AR for evaluating fibre dissection in cadavers and in vivo tractography. By surveying 15 neurosurgeons, neurolinguistics, and neuroscientists, they demonstrated that the overall experience of using AR in fibre dissection was positive (median of 8 out of 10, mean ± standard deviation 8.5 ± 1.4, with 0 being minimum and 10 being maximum). The clinical application of AR fibre dissection was of high value, with a median of 7.0 points (7.0 ± 2.5). AR can also enable making comparisons between disease-free normal anatomy and pathological tissues during the surgery [35]. 92.3% of participants stated that using AR was reflective of their real-life clinical experience. Participants suggested incorporating more anatomical structures, inclusion of fibre tracts from both healthy subjects and patients for comparisons, and developing a more user-friendly setting such as enabling to zoom in/out and rotating the reconstitution could improve the AR setup [35].

Roethe et al. [75] investigated the clinical effectiveness of AR in brain tumour surgery by comparing 16 patients using conventional neuronavigation to 39 patients with AR-navigated microscopes after randomisation. They used MRI, diffusion tensor imaging (DTI), and brain mapping results from navigated transcranial magnetic stimulation (nTMS) in AR to insert information on the surgical field. In the AR group, surgeons used the AR function 44.4% (mean 32.2 min) of the total resection time (mean 72.5 min). In the non-AR group, pointer-based navigation resulted in frequent workflow interruptions (5–28 s each time navigation was used). Pointer utilisation frequency was significantly reduced in the AR group to 2.6 times per resection hour (0–12; SD 2.53), compared to 9.7 times per resection hour (0.8–21.6; SD 5.6) in conventional neuronavigation (P < 0.001). Navigation pointer was mainly used in the AR group for verifying positions during early phases of the surgery and for estimating the brain shift during advanced phases of the surgery. Head-up or head-mounted displays of AR can provide continuous integrated display, enabling pointer-less navigation. Roethe and colleagues also demonstrated that AR is most suited for deep-seated lesions (> 1 cm from the cortex), particularly during targeting small structures, defining the tumour borders, and identification of eloquent areas [75].

Neurosurgeons rated higher the spatial understanding of information (median 3.0 in AR compared to 5.0 in conventional navigation, P < 0.001), the visual accuracy of the overlay (median 3.0 in AR compared to median 5.0 in conventional navigation, P < 0.001), and visual comprehensibility (median 3.0 in AR compared to 4.0 in conventional navigation, P < 0.001), in conventional navigation, due to perhaps familiarity with conventional navigation compared to AR [75]. In contrast, there were no statistically significant differences in the coherence of image fusion (median 3.0 in both groups, P = 0.060) and relevance of visualisation (median 4.0 in AR compared to 5.0 in conventional navigation, P = 0.559), between the two groups. Furthermore, important anatomic structures were blocked completely or partially and were not visible in 43.6% of cases using AR. Despite such challenges, 66.7% of surgeons found AR helpful for their surgical cases, and 76.9% of those cases were categorised as deep-seated lesions [75].

According to Roethe et al. [75], neurosurgeons may have different preferences for the usage of AR during the operation. Neurosurgeons requested more AR information, especially during the initial phase of tumour resection. Also, they were more comfortable with using AR visualisation as guidance for the entire tumour removal. Therefore, personal preferences can affect the usage of AR and VR, and such variations should be considered to promote the usage of such technologies.

Discussion

AR and VR are innovative domains of medicine with growing applications in neurosurgery, including training, patient care, operative planning, intraoperative assessments, and postoperative care [8, 19, 36, 37, 52, 55, 58, 69, 86, 90]. AR and VR can be used during the preoperative phase of AC (i.e. for example, in training residents and medical students, and simulating the operating room experience for patients), in the intraoperative phase (i.e. mapping cognitive functions), and in the postoperative phase (i.e. patient rehabilitation) (Fig. 2).

Fig. 2
figure 2

A schematic summary of AR and VR usage at preoperative, intraoperative, and postoperative phases of AC. AR, augmented reality; OR, operating room; VR, virtual reality

Full size image

While language and motor mapping have been routinely performed during AC, non-verbal and non-motor mappings — such as mapping of the visual field and social cognition — have been limited due to challenges with adapting classical bedside tests to the restrictive environment of the operating room during AC [2, 9, 92]. Such associated challenges make the prediction of postoperative optic visual field defects difficult [84]. Multiple approaches, such as tractography-based neuronavigation, visual evoked potentials, and direct subcortical stimulation, have been suggested for mapping optic radiations [23, 44]. Furthermore, impairments of other non-verbal and non-motor cognitive processes, such as social cognition, including body language, empathy, facial expression, and theory of mind after neurosurgery, can be a source of concern [78]. Patients with postoperative social cognitive impairments can have difficulties in conceptualising and understanding thoughts, beliefs, desires, behaviours, and intentions of other people, and such defects have been mistakenly attributed to impairments in language and executive functions with few studies addressing them [34, 64, 72, 77]. Various tests can be designed and implemented using VR to assess domains other than language and motor during AC.

Current state of using AR and VR in neurosurgery

VR generates realistic images and sounds to simulate users’ physical presence in a virtual environment to develop new tests for evaluating neurological functions in AC, for example by using avatars mimicking humans [9]. Using avatars to interact and socialise in VR has been reported previously in cognitive neuroscience [27]. Participants perceived avatars similar to humans and connected with them as they did with humans in the real world because social norms in the VR environment, including gender, personal space, and eye gaze, are governed by similar rules to the physical world; therefore, making their usage for non-verbal cognitive processes, such as empathy and theory of mind possible [4, 94]. Multiple other tasks testing other domains, such as auditory association, calculus, working memory, praxis, and gnosias, can be assessed using VR [46]. For example, patients can be asked to hear an animal’s sound and name an animal to assess the auditory association and perform arithmetic tasks to assess calculus, working memory, and attention [46]. We suggest that the development of future interactive VR environments can assess different domains within the same task to facilitate the preservation of multiple eloquent areas. Furthermore, VR enables investigating multiple variables such as age, race, sex, and expression, which cannot be tested in a physical world during AC.

Multiple AR-based approaches, such as image projection, head-up display (i.e. transparent displays that present data without requiring users to look away from their viewpoint), head-mounted display (i.e. devices which are worn on the head or as part of a helmet), monitor- and tablet-based visualisation, and image insertion into the surgical field, have been investigated [3, 13, 14, 19, 30, 39, 47, 49, 62, 73, 81, 93].

AR in neurosurgery involves an overlay of the necessary information, such as CT/MRI and functional information, augmented onto the surgical field [63]. Thereby, AR can provide crucial information such as tumour boundaries and risky adjacent structures to reduce surgical risks, reduce intraoperative cognitive load, and provide information for the entire surgical team [19, 37, 47, 75]. Multiple factors have been reported to influence the suitability of cases for AR and VR. Previous investigations have shown that projection techniques are best suited for superficial brain tumours [37], whereas head-up AR techniques are suitable for small deep-seated lesions [45], despite a reduction in depth accuracy [60].

Challenges and future direction of AR and VR in neurosurgery

There are some challenges associated with AR and VR use in neurosurgery, including registration errors, depth perception errors, temporal asynchrony in visual and tactile modalities, and blocking of visual fields [30].

The quality of AR visualisation is significantly dependent on the quality of the CT and MRI data as well as successful image fusion [43, 75]. Resolution of the inserted image and segmentation is particularly challenging in neurosurgery as small diameter vessels and nerves are poorly delineated in original data [75]. Therefore, challenges associated with demarcating anatomical structures are not solved using AR. From the neurosurgeons’ perspective, pointer-free navigation during tissue preparation and tumour resection is the most suitable feature associated with AR and VR [75]. Using AR and VR can eliminate having alternative views of the surgical field to combine physical and virtual information. One of the challenges in accuracy of neuronavigation is the brain shift during AC which is present in both conventional and AR-based navigations. This can be addressed by using intraoperative imaging and updating the feeding data to AR [51].

Fade-in display of surgical information and pointer-free navigation may pose some challenges, such as completely or partially blocking the view of the surgical field and impairing depth assessment. One study demonstrated that most otolaryngology expert participants preferred the display of information in the periphery over AR visualisation in the focus level due to the distraction effect and visual occlusion [22]. Other restrictions associated with AR and VR technology include an overlay of working and viewing fields, the lack of 3-D depth in the AR, and visualisation offsets caused by MRI data resolution. One of the common problems associated with AR was 2-D versus 3-D visualisation of surgically relevant information during the surgery [75]. AR utilisation can be disruptive and challenging, especially for inexperienced users, as surgeons require training to read and apply relevant information [75]. To overcome these challenges, future studies can focus on designing user-friendly interfaces to minimise the mental and visual distractions for the surgeons. Despite these challenges, 3-D displays of various datasets over the surgical field and advances in robotic surgery, registration, and display hold promise for the use of AR and VR in the operating room in neurosurgery.

Furthermore, mapping more complex cognitive functions, such as emotions in AC, ideally requires preoperative screening to establish the baseline characteristics of the patient as well as postoperative monitoring and rehabilitation [46]. It has been shown that AC patients experience less stress, anxiety, and depression in different phases of the operation [67]. We suggest that VR can be used to assess the intraoperative level of anxiety and stress and reduce them, for example by having the patient interacting with virtual avatars.

Limitations

This systematic review is subject to some limitations. AR and VR in AC are relatively novel concepts reflected in the limited number of publications. Our review was limited to papers published in English, and there was heterogeneity in the articles reviewed. The patient numbers were low, and the number of participants in each study and data collection period varied widely. AR and VR technologies are not widely used in all neurosurgical settings, especially in countries with resource restraints. This was reflected in the fact that all studies included in this review were from high-income countries. Furthermore, participants in included studies were patients and surgeons, and the studies reviewed had different inclusion and exclusion criteria. No study investigated both AR and VR in the same group of participants. Studies focused on different phases of AC, namely preoperative and intraoperative, and no study was done at the postoperative phase of AC. Therefore, future studies are required to evaluate AR and VR at all phases of AC. Despite such limitations, the current review can be a useful addition to understanding AR and VR in neurosurgical procedures and prompt further research.

Conclusion

The technologies concerning AR and VR are rapidly advancing in neurosurgery, and they can be used at different phases of AC by patients and surgeons. As the technological capability of AR and VR advances, neurosurgeons are finding more applications in training and patient interaction, within and out of the operating theatre. AR and VR have been tested against existing protocols and have been shown to generate equal or better outcomes in some studies. While the benefits of AR and VR certainly outweigh their risks, multiple factors, such as case selection, technology availability, visualisation, and technical limitations, should be considered when using AR and VR in the neurosurgical operating theatre. Results from studies included in this review can only be indicative of certain domains in AR and VR, and further technical improvements are required to eliminate existing problems with AR and VR. Development of future AR and VR environments can be customised for patients and surgeons to assess specific networks and domains that they desire to be preserved. In addition, more training is required to ensure competencies are achieved among neurosurgeons and trainees. Future prospective multi-centre studies with long-term follow-up and larger sample sizes are required to assess patient outcomes in AR and VR compared to conventional approaches.