Multimodality image fusion is an image processing procedure that allows the spatial alignment of multiple images acquired from one or more imaging devices and the visualization of the fused information within a single display or representation. The rationale behind the development and use of an image fusion strategy in a clinical environment resides in the fact that different images can synergistically contribute to a more comprehensive characterization of the disease and provide guidance for the most appropriate medical or interventional treatment.1,2,3 More than that, while traditionally physicians have been trained to mentally integrate different imaging tests to provide the best patient’s care, it has been widely demonstrated how a fused display can be far more informative and diagnostically powerful than a mental representation.4 Nowadays, applications for medical image fusion range from oncology and radiotherapy, to neurology, to image-guided surgical guidance, to cardiology. Over a decades-long period images from a large variety of imaging modalities have been used to develop image fusion frameworks—invasive and noninvasive, anatomical and functional—prompting the proliferation of hybrid devices, such as PET/CT and PET/MR, SPECT/CT, CT/MR.5,6,7

Specific to the cardiology field, image fusion has been primarily used to integrate anatomical datasets, such as CT, MR or fluoroscopy, to functional ones, such as SPECT or PET. For a number of cardiovascular diseases, anatomy and physiology represent two sides of the same coin: assessment of both can greatly improve physicians’ ability to diagnose and plan treatment. Coronary artery disease (CAD) assessment is greatly improved by the availability of information on the anatomy of coronary vasculature, the degrees of a stenosis or the presence of diffuse disease, the extent of myocardial ischemia.8,9,10,11 The initial attempts to fuse anatomical and functional information date back to the early 1990s12 and successively became more established techniques that proved the incremental diagnostic power of fused imagery compared to single modalities or side-by-side interpretations.13,14,15 In cardiac electrophysiology, the success rate of interventions such as cardiac ablation16 and cardiac resynchronization therapy (CRT) strongly relies on the ability to not only accurately select patients, but also to correctly plan the procedure and, in case of CRT, to deliver the pacemaker leads into optimal position.17

Nevertheless, despite extensive research and test applications, multimodality image fusion techniques have not been translated into clinical routine. The main reason is that image fusion remains a highly technical procedure at the conjunction of several scientific disciplines beyond medicine, such as image processing, computer vision and pattern recognition, and with a number of hurdles still to overcome. Primarily, image registration techniques18 can suffer from significant inaccuracies related to patient positioning and tissue non-rigid deformations. In situations of relative organ rigidity, such as with the brain, image registration can be both automated and accurate. But in cardiovascular applications cardiac motion and respiratory artifacts, not to mention patient motion during long acquisition times, can make rigidity assumptions inadequate and results suboptimal. On the other hand, techniques for deformable registration19 are not established yet, mostly manually driven and can be time inefficient.

The article by Massera et al20 in this issue of the Journal represents a constructive step forward towards enhancing current image registration techniques and provide access to physicians to user-friendly image fusion tools. The authors presented and tested their integrated software FusionQuant for 18-F Sodium Fluoride PET and CTA image fusion in the context of the study of aortic stenosis and assessment of leaflet calcifications. A major issue when fusing nuclear acquisitions to anatomical ones resides in the fact that PET/SPECT studies are acquired over a number of heart beats while a CT represents a single snapshot of the cardiac cycle. Routinely, an averaged nuclear image or only a small portion of the nuclear gated data is used for fusion purposes, likely reducing accuracy. Instead, in their work the authors integrated into the FusionQuant software a deformable image registration algorithm that uses the entire gated acquisition and compared their results to another software package with only rigid image registration capabilities. The more sophisticated image registration approach resulted in decreased image noise, increased SNR and improved scan-rescan reproducibility of the measured indexes. Another important feature of image fusion, which is also highlighted in the work of Massera et al,20 is in fact its ability to investigate over-time disease progression or treatment outcomes by fusing follow-up images to previous acquisitions in an objective and consistent manner. Importantly, these enhancements came without impacting usability, as only simple manual interactions were required, and a decrease in total image processing time.

The key to enter a clinical environment remains undoubtedly automation. Clinicians do not have the time for prolonged image-processing tasks and possess little expertise in assessing the efficacy of complicated registration algorithms. In this field developers of multimodality image fusion frameworks and applications have still some work to do. The nuclear cardiology community has witnessed the tremendous impact that software packages for automated and robust image quantification have had in the clinical world. Image quantification, and particularly image segmentation, is less established for anatomical acquisitions. The normal variability in anatomical structures in addition to variations associated with pathological conditions make the design of general algorithms particularly challenging. And yet they are the more necessary, as the number of images per dataset have moved from a few hundreds to thousands. The recent introduction of machine learning (ML) techniques has transformed many areas of medical image processing with important developments also for the assessment of cardiovascular diseases.21 Despite some reasonable concerns on the use of ML techniques without experts’ supervision, there are fields of image processing that could greatly benefit from the availability of automated and reliable algorithms for image quantification.

Multimodality image fusion is also by definition cross-specialty. In addition to the further technical developments necessary in image processing methods, an environment of collaboration and communication between the different departments and radiology specialties will be fundamental. Image fusion applications will need to seamlessly access raw data, complete their processing tasks and enabling visualization possibly from a single workstation. Concurrently, physicians may encounter a learning curve in approaching multimodality image fusion frameworks as they will be asked to interpret images from nuclear and anatomical imaging and to use new visualization techniques and hybrid displays.

It is also worth addressing two main concerns that are commonly raised against the use of multimodality image fusion for diagnostic purposes: the appropriateness of performing multiple imaging tests and the radiation burden associated with them. Promoting the development of more sophisticated tools for image fusion and their utilization in clinical environments does not equate to support an unchecked use of imaging resources and possibilities. In fact, while in many instances the first test allows to properly diagnose the patient with no need for additional imaging, there are controversial cases for which frequently more than one exam is administered before a final diagnosis can be made. Multimodality image fusion—if available—represents the best way to take full advantage of the information already obtained, potentially reducing the need for further assessments or interventions down the line. On the issue of patient radiation exposure, pressure from the public and the medical communities has contributed to a renewed effort from physicians and imaging companies towards decreasing radiation doses for patients. The nuclear cardiology community is promoting guidelines22 that encourage departments to decrease radiation burden and develop new technologies to minimize injected doses and acquisition times. In cardiac CT, specific protocols have been designed to perform clinical-quality acquisitions with a minimal radiation burden. A compelling case for multimodality image fusion could then be made particularly for certain populations. For instance, in CAD assessment, a significant percentage of patients scheduled for elective coronary angiography ultimately do not exhibit obstructive disease.23 A moderate increase in radiation dose during initial diagnosis could prevent unnecessary procedures. In complicated surgical interventions, a treatment plan could be outlined beforehand reducing procedure time while improving outcomes.

Although it is generally accepted that images from complementary modalities provide incremental prognostic and diagnostic value, in the PACIFIC study Danad et al24 investigated the accuracy of major noninvasive techniques (functional, anatomical and hybrid) for ischemic heart disease assessment and reported no additional diagnostic value for hybrid imaging, either by means of SPECT/CTA or PET/CTA. While differences in methods for image quantification may be one of the reasons for these findings, the study underlines the existence of a dynamic and not completely understood relationship between anatomy and function, in this case between the degree of a stenosis and the indication of ischemia. Robust multimodality image fusion techniques by investigating different aspects of a disease can certainly help us unravel some of these mechanisms.