Introduction

Structural heart disease has been defined as non-coronary cardiac abnormalities encompassing both congenital and acquired defects [1]. Complex spatial relationships and significant individual anatomical variation make planning catheter and surgical interventions challenging, both for initial, and subsequent procedures. Detailed planning using imaging data from echocardiography, computed tomography (CT) and magnetic resonance imaging (MRI) is the standard of care and has traditionally been based on two-dimensional (2D) planar reconstructions which aid the user to mentally reconstruct the patient’s anatomy. As the complexity of surgical and transcatheter intervention has increased so has the need for more advanced image display methods which give the user a better, more intuitive understanding of the patient’s intra- and extracardiac spatial arrangement.

Initially, volume- or surface-rendering techniques were used to create an illusion of depth within a 2D image using colour and lighting effects. In the past 15 years three-dimensional (3D) printing technology has enabled the creation of a 3D physical replica of the patient’s heart from cross-sectional imaging data with the ability to simulate procedures. However, the significant time and expense of creating models, the difficulty in segmenting finer cardiac structures such as valves, and inability to recreate dynamic motion, are key limitations. Extended reality (XR) data has the potential to provide accurate and detailed 3D image visualisation of multimodality imaging without these constraints.

This review summarises the current range of reported clinical applications and validation of XR technology with respect to procedural planning in structural heart disease, highlights the current limitations in both the technology and the current literature, and discusses scope for the future.

Extended reality - definitions and technology

XR is an umbrella term that covers virtual reality (VR), mixed reality (MR) and augmented reality (AR) [2, 3]. The XR ‘spectrum’ is illustrated in Fig. 1. In this review, based on the publications analysed, VR will denote technologies where the real world is completely occluded and replaced with a virtual input, MR when virtual objects can interact with the user and the real-world environment, and AR when virtual information is visible but cannot respond to the user or environment.

Many XR visualisation systems are headsets that achieve 3D image display through stereoscopy, creating the impression of depth in the viewed image [4]. VR headsets are opaque, obscuring the outside world. Conversely, MR and AR headsets must show the user the outside world, and may achieve this in one of two ways. First, they may be transparent, allowing the user to see through to the real world. An image of the virtual objects is then superimposed upon this, for instance using optical waveguide technology [5]. Alternatively, they may use an opaque headset with cameras to provide a live video ‘feed’ of the user’s real surroundings, with virtual objects incorporated to create a merged environment. In VR and MR systems user interaction may be provided either by dedicated controllers, or by using hand and gaze tracking combined with gesture control.

Non-headset XR systems include stereoscopic flat monitor displays. These may require the user to wear special glasses, to provide an image with depth, and track their head position to provide an immersive window into a virtual world. Smartphones and tablets may also be used as AR systems, superimposing virtual objects upon the live video feed from their cameras. Digital holography, which differs from the stereoscopic displays described above, uses an ‘over the head’ display to project an interference-based volumetric hologram suspended in free space [6].

Fig. 1
figure 1

Pictorial representation of major XR modalities and key features

Full size image

Methods

A systematic search methodology was utilised in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines [7]. A literature search was performed using Medline, Embase and the Cochrane library using a combination of search terms including all publications from inception until November 2021. The Medline and Embase search strategy are detailed in supplementary table S1 and was adapted for the requirements of the Cochrane library.

After removal of duplicates, the title and abstract of each publication were reviewed to determine eligibility. Only publications written or translated into English which reported first-hand on results of a study investigating use of XR in diagnosis, procedural planning or guidance of structural heart diseases were included. Specific exclusions were (1) non-primary research or review articles; (2) subspeciality fields to which the search was not adequately directed, e.g. transplant, electrophysiology, ischaemic heart disease; (3) duplicate publications, in which case only the study reporting the largest dataset was included. The approach to identification of relevant papers is illustrated in Fig. 2. Included articles underwent full text review, and references cited in these studies were also examined for relevance using the same inclusion criteria. Full text publications of relevant studies were reviewed by one reviewer (N.S.). Two other authors (J.M.S. / K.P.) arbitrated on inclusion or exclusion.

Fig. 2
figure 2

Flow chart demonstrating systematic literature search with inclusions and exclusions

Full size image

Results

56 studies were included in the final analysis. The included publications fell into four main categories – XR for planning cardiothoracic surgery, transcatheter interventions and device-sizing, intra-procedural guidance and diagnostics.

Publication trends

Publications by year

Between 2005 and 2017, 12 studies were published, with up to 2 publications each year during this time period. From 2018 until the end of 2021, 44 studies were published in this area, more than the total from the preceding 13 years. The number of annual publications has increased each year since 2017; from 6 to 2018 to 16 in 2021. This trend of rapidly increasing research interest is shown in Fig. 3.

Fig. 3
figure 3

Bar chart demonstrating included publications by year and type of use

Full size image

Types of XR used

VR is the most studied form of XR, with 28 included studies (50%) describing its use. MR also features prominently, with 22 publications (39%) using this modality, mostly published after 2018. There is clear predominance of MR and AR in real-time guidance of cardiac procedures, used in 93% of these studies (Fig. 4). Publications using XR for pre-procedural planning of catheter interventions also used MR systems most frequently (60%). In contrast, VR was the dominant modality used for surgical planning (58%) and diagnostics (67%).

At least 33 distinct XR systems were used, with some being used in more than one study. Seven studies used Echopixel’s True 3D Viewer, 5 Carnalife Holo, 3 Artiness Articor, two Vesalius 3D Stereo Viewer, two CardioVR and two used Barco i-Space. To the best of our knowledge, five XR systems had definitive approval as medical devices by major regulatory bodies at the time of writing (Echopixel’s True 3D Viewer, Carnalife Holo, Artiness Articor, Vesalius 3D Stereo Viewer and Realview Imaging).

Fig. 4
figure 4

XR subtypes used in different study categories

Full size image

Study methodology

Twenty-six prospective studies were included in this literature review, accounting for 46% of all publications. Four of these prospective studies were comparative, comprised of one randomised-controlled study, 2 non-randomised control studies and one comparative study with no control group (Fig. 5). Twenty-two (85%) of the prospective studies were descriptive case reports or series. In the remaining 23 studies, a retrospective comparative methodology predominated, whereby XR visualisation was compared to standard visualisation of imaging on a 2D screen, to other 3D technologies or intra-procedural findings.

Fig. 5
figure 5

Graphical representation of study methodologies utilised in included studies

Full size image

Applications

XR for planning cardiothoracic surgery

Eighteen studies assessed XR for planning cardiothoracic surgery. Publications with 5 or more patients are summarised in Table 1.

Two experimental studies included a control group. Gehrsitz et al. evaluated the utility of photorealistic (cinematic) rendering displayed in MR in 26 patients undergoing a range of congenital cardiac surgeries, compared to retrospectively matched controls who had undergone surgery without MR review [8]. Intra-operative preparation time was significantly shorter in the study group (mean 58 min) compared to controls (mean 73 min). Surgeons preferred MR visualisation in most domains, but found it less useful for intracardiac anatomy, which the authors reported was due to the specific rendering technique. Ye et al. conducted a single-centre randomised study comparing XR to standard image visualisation in 34 patients with double-outlet right ventricle (DORV) [9]. The intervention group, where surgeons reviewed imaging in MR, had a significantly shorter planning time (mean 52 min) compared to the control group (66 min). Every patient in the MR group had their anatomy correctly identified pre-operatively with no change to the proposed surgical plan, whereas the surgical plan was altered in 3 control group cases due to unexpected intra-operative findings.

In other studies, benefit of XR was evaluated by surgeons reviewing imaging both virtually and on standard flat-screen software. Lu et al. asked surgeons to review data in MR as well as on 2D screens prior to performing a range of congenital cardiac surgeries [10], which led to alteration in surgical strategy in 2 cases (8%). Pushparajah et al. compared VR to flat-screen imaging review for planning atrioventricular (AV) valve surgery in 15 patients [11]. In 67% of cases, surgeons reported that VR gave them more confidence in the anatomy and they would have modified their surgical approach in nearly 60% of cases. Milano et al. found that surgeons could better delineate patient anatomy and provide an accurate surgical plan after reviewing DORV patients’ imaging in VR, compared to 3D-printed models or a 3D segmentation on a flat screen [12]. Vettukattil et al. used MR image visualisation to assess 7 patients’ suitability for biventricular repair after previously being considered either inoperable or suitable only for univentricular palliation [13]. All patients successfully underwent biventricular repair with no mortality in a mean follow-up period of 22 months. Pather et al. compared VR to 3D prints and flat screen review [14], with structured feedback suggesting that clinicians assessed 3D prints and VR as superior to 2D visualisation and VR had the most potential benefit for surgical planning.

The majority of prospective studies were case reports or series describing feasibility of XR for surgical planning and obtaining structured feedback from surgeons on perceived usefulness of XR. Larger case series included XR for planning surgery in 5 cases of pulmonary atresia and major aortopulmonary collaterals (MAPCAs) [15], complex aortic valve reconstruction in 26 patients unsuitable for valve replacement [16], 17 pulmonary artery reconstructions and unifocalisations [17], and 6 complex redo and minimally invasive surgeries in adults [18]. In seven case reports XR was used for planning repair of DORV [19], truncus arteriosus [20], a complex ventricular septal defect (VSD) [21], right ventricular to pulmonary artery conduit stenosis and left ventricular outflow tract (LVOT) obstruction [22], tetralogy of Fallot (TOF) [23], and resection of a large cardiac tumour [24]. In all studies, physicians and surgeons reported subjective benefit of XR visualisation for surgical planning. Although not yet trialled clinically, Vigil et al. described a workflow for VR intracardiac baffles which could be inserted within MRI-derived models, and which the authors postulated may be beneficial for patients with DORV undergoing complex biventricular repairs [25].

Table 1 Studies assessing XR for surgical planning (excluding case series/reports with fewer than 5 patients)
Full size table

XR for planning transcatheter intervention and device implantation

Four studies assessed XR image visualisation for planning catheter interventions (Table 2). Four case reports and 3 proof-of-concept studies were also identified. A large experimental study compared CT-derived segmentations visualised in MR to traditional procedure planning using 2D transoesophageal echocardiographic (TOE) to plan transcatheter left atrial appendage occlusion (LAAO) [27]. There was a statistically significant reduction in total procedure time and device wastage, with a trend towards lower contrast use in the XR group. Two retrospective studies assessed XR for sizing of valves or devices. Sinha et al. found no significant difference between MR-derived annulus measurements and implanted valve size in 38 patients who had previously undergone TAVI, and concluded that MR may improve landmark identification and assist sizing of valves [28]. Another group used a VR system to predict mitral and aortic paravalvar leak occluder sizing [29], and although they found device size most closely approximated measurements made on 2D CT images, feedback was that VR was the most helpful tool for assessing the 3D nature of the leak. Tandon et al. used segmented CT datasets of 28 patients with superior sinus venosus atrial septal defect (SVASD) in VR to determine suitability for transcatheter closure [30]. They incorporated cylindrical stent models into the system which enabled assessment of potential obstruction of anomalous pulmonary veins.

The case reports described use of XR for successful planning of a challenging SVASD case [31], transcatheter Fontan completion [32], systemic venous baffle obstruction post-atrial switch [33], and hybrid approach in multiple muscular VSDs [34]. Three further studies described workflows and feasibility for device incorporation or sizing in LAAO [35], and septal occluders in VSDs prior to transcatheter intervention [36], but were not tested clinically in these publications.

Table 2 Studies using XR to plan catheter intervention and device implantation (excluding case series/reports with fewer than 5 patients)
Full size table

XR for intra-procedural guidance of cardiac interventions

Larger studies assessing XR for intra-procedural guidance are summarised in Table 3. Prior to use in humans, a number of systems which use XR guidance with catheter electromagnetic tracking systems have been tested in non-human models. James et al. demonstrated superiority of their VR system for guiding transseptal puncture in a cardiac phantom compared to fluoroscopy [37]. Another group published a series of feasibility studies testing AR intra-procedural guidance for minimally-invasive cardiac surgery in phantom and animal models. In a cardiac phantom, they demonstrated similar precision of AR and fluoroscopy for TAVI valve deployment by a single operator [38]. The system has also been tested in a live porcine model to demonstrate feasibility of ultrasound-guided minimally invasive mitral valve replacement[39], atrial septal defect (ASD) closure [40, 41], and neochordae prosthesis insertion for flail mitral leaflets [42], with small studies showing that AR guidance could reduce procedure time and the risk of injury to cardiac structures by improving navigational accuracy.

At the time of writing, there are no controlled experimental studies using XR for intra-procedural guidance in cardiac structural interventions. A proof-of-concept study demonstrated that an MR guidance system enabled placement of cerebral embolic protection (CEP) devices in 6 patients undergoing TAVI [43], eliminating the need for aortic arch angiography prior to filter placement. Another group published case reports describing use of their MR system for intra-procedural guidance in a range of cardiac catheter procedures including patent ductus arteriosus (PDA) device occlusion[44], TAVI [45], pulmonary artery interventions [46], balloon mitral commissurotomy [47], and LAAO [48].

Table 3 Studies using XR for intra-operative guidance (excluding case reports or series with fewer than 5 patients)
Full size table

XR as a diagnostic tool

Fifteen studies assessed XR for its diagnostic capabilities for structural heart lesions, nine of which included more than 5 patients (Table 4). The most frequent application was for assessment of AV valve morphology and function. A Dutch group performed 2 small studies assessing an early VR system to assess AV valve function [49, 50]. They showed that it was possible to differentiate normal from abnormal valves, and appreciate additional information in VR which had not been delineated on standard 2D assessment. Beitnes et al. assessed diagnostic yield of VR in mitral valve prolapse [51], and demonstrated > 90% sensitivity and specificity compared to intra-operative findings. In addition to assessing anatomy, XR was also assessed for measurement accuracy and precision. Measurements in echocardiographic data from normal and diseased mitral valves were assessed in an industry-standard semi-automated mitral modelling system and in MR [52]. Overall, agreement between systems was better for normal than pathological valves, which the authors attributed to the loss of normal valve architecture making measurement more challenging. Another group assessed measurements on 3DE images of a phantom and AV valves in VR against two industry standard platform [53]. VR was determined to be more accurate but less precise than standard software, the latter attributed to lack of familiarity with their system. Two other publications demonstrated that VR display reduced AV valve measurement variability and improved repeatability compared with standard software [54, 55].

XR visualisation was also assessed for the identification of septal defects [56, 57], DORV [56, 58], MAPCAs[59], precise aortic root anatomy [60], and complex congenital lesions [61]. In all studies they were either equivalent or superior to standard methods of image display for detection of landmarks or lesions. In one study, the interpretation time was also significantly reduced [59]. Kim et al. compared VR and non-immersive displays for CHD diagnosis in cardiology trainees, determining that accuracy was higher using immersive VR and that this bias increased with more complex lesions [62]. Two publications also provided a comparison of XR display to 3D-printed models: Raimondi et al. found that a senior cardiac surgeon’s diagnostic interpretation of complex CHD cases was equal or better using VR compared to a 3D printed model [61]. Based on structured feedback from 35 participants, Lau et al. concluded that 3D printed models and VR projection of segmentations were similarly effective in conveying anatomical information [63].

Table 4 Studies using XR for diagnostics including measurement (excluding case reports or series with fewer than 5 patients)
Full size table

Discussion

Interest in XR for procedural planning in structural heart disease is rising, with a rapidly expanding literature base. The surge in research interest since 2018 mirrors XR technological improvements and increased availability over this time period. The launch of the recent generation of VR headsets and powerful graphics cards in 2016 significantly improved user experience via high quality images, realistic user interaction and reduced motion sickness. This set the scene for greater interest in the XR industry, with market projections estimating an increase in global market value from USD 11 billion in 2021 to 227 billion by 2029 [64]. Large industry players are investing heavily in XR, which will likely lead to greater accessibility, affordability and user-friendliness of equipment, and in turn increase its uptake in healthcare.

Review of the literature has indicated different use profiles for VR, MR and AR. VR continues to be the most-used type of XR at present in 50% of studies, followed by MR in 39%, AR in 9% and digital holography in 2%. VR was the dominant modality in surgical planning and diagnostics, probably because it completely immerses the user in the virtual environment, which may be advantageous for detailed image interrogation. Nonetheless, the enclosed headset isolates the user from their surroundings, and we found that studies using XR for real-time guidance strongly favour AR and MR, which enable the user to maintain sight and awareness of their real-world surroundings. Digital holography may also be applicable for live procedure guidance, although its dependence on bulky boom-mounted equipment makes it less practical.

Whilst holding much promise, AR and MR devices continue to have significant limitations such as a narrow field of view, limited battery life and suboptimal resolution (screen door effect), and technical improvements will likely be required before they are widely implemented in the clinical environment. In addition, an issue facing all current XR modalities is the lack of ability to share the full XR experience with a wider group of viewers without the requirement for multiple headsets. Whilst 3D screens and projectors hold some promise, the technology requires further development to enable the XR experience to be shared in complex case reviews or surgical conferences, which are mainstays in the multidisciplinary care of patients with structural heart disease. A further obstacle which may limit widespread adoption of XR in procedural planning is its clinical evaluation in high-quality studies. Structural and congenital heart disease comprises a small and hugely variable study population and obtaining an adequate sample size to achieve statistical power is difficult. And these difficulties are reflected in of the included publications. There was only one well-designed outcome study which was able to assess for incremental benefit of XR technology over standard imaging, albeit in a single centre and with small patient numbers [9]. This study demonstrated a reduction in surgical planning time of 14 min and improved pre-operative identification of anatomy using XR compared with flat screen display. Other prospective studies were liable to confounding due to use of different imaging modalities in the XR and standard groups and potential bias from use of retrospectively matched controls [8, 27]. Many studies relied either partly or entirely on subjective feedback from users. The vast majority of studies using XR for structural heart procedural planning were case reports or series, and to date there have been only case reports using XR intra-procedural guidance in humans. None of the included studies were designed or powered to assess for superiority of XR visualisation on patient outcome such as complication rate, length of stay or re-intervention rate. Nonetheless, there are some promising results with respect to improved operator confidence, timesaving, reduced device wastage and contrast use, all of which could improve patient outcomes and reduce healthcare costs.

Based on our results, XR has wide applicability within structural and congenital heart diseases, especially when planning procedures where understanding of complex anatomy is required such as for double outlet right ventricle repair, unifocalisation of MAPCAs or surgical or transcatheter repair of sinus venosus atrial septal defects. In addition to complex congenital lesions, this technology lends itself particularly well to visualising 3D interactions and may have value for planning adult structural interventions such as transcatheter valve implantations, appendage occlusion or intervention on the mitral and tricuspid valves. This has the potential to reduce complications such as compression of adjacent structures such as coronaries, pulmonary veins or the neo-left ventricular outflow tract as well as worsening of atrioventricular valve regurgitation. In future, multi-centre collaborations will be essential to enable clinically meaningful outcome measures to be adequately assessed and further expansion into the larger adult structural field could also enable this.

The majority of studies in this review describe the use of XR by surgeons and interventionists as an alternative or adjunct to ‘flat screen’ review with a radiologist or an imaging cardiologist. Structured feedback from many of the included studies suggests that XR lends itself well to use by this cohort given that the interface with imaging is generally more intuitive and enables tailoring of the individual’s experience [14, 54, 58, 63]. Incorporation of devices or patches into the XR environment may also enable clinicians to “road test” a procedure, reduce device wastage and potentially reduce procedure time or contrast use [29, 30, 36]. In this review a smaller selection of studies included radiologists or imaging cardiologists, however, feedback and results were largely positive. Additionally, implementation of novel volume-rendering and automated segmentation techniques into XR solutions could circumvent time-consuming manual segmentation and reduce workload [65].

Whilst XR is unlikely to replace standard 2D screen review in the clinical workflow, it has the potential to become a valuable tool in the arsenal of imaging specialists who have a crucial role in decision-making for patients. XR is most likely to become an adjunct to assist planning the more complex, high-risk cases where potential benefit is highest. It is likely that adoption beyond congenital heart disease would be required to stimulate technological development and investment by industry. Such potential benefits are not limited to cardiac disease, and many groups in other surgical disciplines such as maxillofacial, orthopaedics, neurosurgery and urology also investigating XR as a planning and guidance tool [66,67,68,69].

A lack of consistent regulatory standards and guidance is a challenge facing roll-out of this technology in the clinical arena [70]. Although international bodies have published standards for image quality assessment for near-eye displays, there are currently no validation standards which cover the heterogeneous and rapidly expanding range of hardware and software used in XR, nor of the interaction between them [71, 72]. A few groups have published their individual workflows and rendering pipelines, opening avenues for peer review and external validation, though this is not common practice [36, 73]. For an XR planning system to become approved as a medical device requires regulatory approval by country-specific authorising bodies, and requires individual review of the device’s safety profile including risk assessment, verification, validation and clinical evaluation data. From our review, only 5 of the included XR systems had definite regulatory approval and indicates that this process is challenging and may not be being consistently applied even in prospective studies impacting patient care. Standardisation of guidance and frameworks by the various authorising bodies could enable a more consistent approach to the evaluation of XR planning systems, and would promote multi-centre studies to evaluate their clinical efficacy and safety.

Conclusion

The potential of XR to assist with planning and guiding cardiac procedures is beginning to be realised, with many researchers focusing on this area as an alternative or adjunct to other 3D technologies. Recent advances in computer power and headset technology mean it is now often compact enough for use in the clinical setting while still providing excellent image quality. This review has shown promising work suggesting that XR can enable intuitive understanding of 3D anatomy, and is a feasible option to assist with planning and guiding complex cardiac surgeries and interventions. Further work is required to demonstrate patient benefit or superiority to existing image visualisation systems. Given the increasing range of hardware and software options, as well as ongoing technical innovation, frameworks and standards from regulatory bodies could assist developers to align validation and clinical evaluation efforts and lead to more widespread adoption.