I recently attended the 33rd European Society for Radiotherapy and Oncology (ESTRO) conference in Vienna, Austria (along with approximately 5,000 others). While watching the scientific talks, which included an excellent Emmanuel van der Schueren Award Lecture from Prof. David Thwaites (University of Sydney), I was struck by the number of opportunities for medical image analysis in this field. In this editorial I’d like to briefly describe and highlight some of these opportunities.

Medical image analysis is an exciting field that combines aspects of computer science, physics, biology, and mathematics for diagnosis, staging, treatment planning, and research. Image modalities have a variety of dimensionalities (generally 2D, 3D and 4D) and involve analysis of x-rays, digitally reconstructed radiographs (DRRs), electronic portal imaging (EPID), computed tomography (CT), cone beam CT (CBCT), magnetic resonance imaging (MRI) and positron emission tomography (PET). Different image modalities are often combined. Some example medical image analysis projects from our Biomedical Imaging Group at the Commonwealth Scientific and Industrial Research Organisation (CSIRO), located at the Royal Brisbane and Women’s Hospital, range from automatic knee cartilage thickness estimation from MRI [1]; early detection of Alzheimer’s disease using imaging biomarkers [2]; early prediction of treatment response in advanced gliomas using PET [3], quantitative analysis of brain white matter using diffusion MRI [4]; automatic prostate segmentation [5], and automatic quantification of spine intervertebral disc degeneration from MRI [6].

The three main components of medical image analysis are image registration, segmentation (or auto-contouring) and image reconstruction. Reconstructed images may also require pre-processing, such as noise reduction, artefact removal (such as metal in CT, or correcting intensity inhomogeneity in MRI), or correcting CBCT intensities to enable CBCT-based dose calculations (by segmentation of tissues, or deformable registration). The DICOM-RT standard enables the transfer of contours between commercial treatment planning systems and external software for registration and segmentation.

Image registration involves finding an optimal transformation between two images. This transformation can be optimised using either the similarity metric between image intensity (using normalised mutual information, correlation, or mean squared difference) or features (distance between points or surfaces). The image transformation can be either global (rigid or affine, allowing translation, rotation, scaling and shearing); or non-rigid (often called deformable or non-linear) where the change in position of each voxel in the moving image is defined by a deformation vector. Two of the most frequently used non-rigid registration algorithms are free form deformation [7] and the non-parametric demons algorithm [8]. The most accurate type of registration algorithm (and parameters) to use depends on the application. Different methods may be required for mono-modal (e.g. between CT scans) or multi-modality registration (e.g. MRI to CT). When contours are available on both the moving and target images, it is possible to improve registration results using structure-guided registration methods [9].

One area of active research is to validate the accuracy of these algorithms, particularly in commercial software. Spatial accuracy can be assessed with physical phantoms (one talk from ESTRO described the use of ex vivo porcine bladders), contour propagation between registered images, and registration errors between known landmark points. Work presented at ESTRO included CT-MR registration for prostate treatment planning, ex vivo pathology to in vivo registration (for cervical and prostate cancer treatment), and calculating dose accumulation (dose can be treated as an image and deformed using previously calculated deformation fields between images acquired at different fractions). Image registration is also important in voxel-wise predictive models of toxicity [10].

Automatic image segmentation (or auto-contouring) is also an active research area which involves separating an image into regions. Types of algorithms range from simple thresholding, classification (e.g. for tumour or scarring delineation), region and edge based algorithms to model based methods. In radiation therapy, segmentation is typically applied to identify target structures and organs at risk. Segmentation is a difficult problem due to non-isotropic voxel spacing, partial volume effects, poor contrast, noise, artefacts, and missing boundaries.

Atlas based segmentation involves the use of one or more contoured scans in an atlas training database. In atlas based segmentation these atlases (and contour labels) are registered to a new target image. Depending on the atlas scheme used, one or more registered labels are selected and fused to generate automatic contours. Determining how to select and combine the most similar atlases from a training database is an active research area. The number of atlases in the training dataset has an impact on segmentation results, and there is a need for these atlases to adequately represent anatomical variability in the target population.

Deformable models are also frequently used for image segmentation. A common approach uses contours from different patients which are converted to surface meshes (with the same number of landmark points) and aligned. Principal component analysis can be used on these points to identify the main modes of shape variation. In the active shape model approach intensity profiles at each point are also included. To segment a new image, the model is initialised, often automatically with an atlas based method, and allowed to deform to find the best matching set of intensities for each point. The main advantage of the method is that segmentation is constrained to anatomically realistic shapes. Statistical shape models can also be used for population based models of non-rigid anatomical deformation (for example to quantify and predict inter-fraction geometric uncertainty) and for predictive models for adaptive planning. Combined with an atlas/registration approach these models can be applied in dose accumulation and construction of plan libraries (generating a variety of plans to account for predicted organ variations, for example in the bladder).

There are also research opportunities in comparing the accuracy of both manual (inter- and intra-observer variability) and automatic contouring approaches. Common metrics for evaluating contours include centroid or point distances; voxel-wise overlap measure such as the Dice similarity coefficient, Concordance and Jaccard Indexes; and surface distance measures such as mean absolute surface distance and the Hausdorff distance. Inter-observer contouring differences are still the major source of systematic error in radiotherapy treatment [11] and there are research opportunities in quantifying and reducing these differences. Inter- and intrafractional organ differences including shrinkage and motion can also be evaluated in the same manner (for example by generating CTV surfaces and calculating surface distance measures).

Intra-fraction tracking and motion management are research areas which may involve medical image analysis. Sample include automatic gating, motion compensated CBCT to reduce blurring from breathing motion, and 3D motion estimation from MRI for abdominal tumour treatment. These often require fast registration, segmentation and tracking.

MRI-alone treatment workflows and MRI-linac systems provide a range of exciting research opportunities. Viewray presented results from their commercial MRI-linac system at ESTRO. Research topics include MRI simulator setups, the generation and validation of pseudo-CT (also known as substitute-CT) scans from MRI, the use of new MRI sequences for more accurate tissue classification and the quantification and correction of MRI geometric distortion.

Advances in molecular imaging modalities, PET in particular, enable exciting opportunities including measuring treatment response to provide guidance for adaptive treatment, localization of recurrence enabling focal salvage treatment, dose painting and spatially measuring biological and metabolic processes (such as hypoxia). PET can be further enhanced through kinetic analysis of dynamic list mode or serial acquisitions, in addition to multi-tracer or multimodality imaging [12].

The combination of MRI and PET in the same machine will enable high quality motion registration and compensation (motion is a major factor in degrading image quality in PET, as PET images are usually averaged during sessions in excess of 30 min). Attenuation correction requirements for PET/MR are similar to pseudo-CT generation in MR-alone treatment planning.

Other research areas impacting on medical image analysis include the use of GPU computation, health economics, radiomics (using structured data to generate descriptive and predictive models which can identify relationships between image features to phenotypes or gene–protein signatures [13]), and rapid learning (applying semantic interoperability of data to enable data comparison from different trials/centres for decision support [14]).

Finally, there are a large number of free, open source, tools available for medical image analysis. The Insight Segmentation and Registration Toolkit (http://www.itk.org/) is a mature and extensive library providing a wide range of algorithms. Cross platform tools built using ITK include: itkSnap, Mevislab, Elastix, milxView, Slicer and Plastimatch. The excellent Visualisation Toolkit (http://www.vtk.org/) also maintained by Kitware has been used in a range of medical image viewers including Paraview, VV, MilxView and Osirix. There is increasing availability of public domain medical image data and contours listed on http://grand-challenge.org/All_Challenges/. These datasets enable onsite challenges, the objective comparison of new methods and reproducibility of published results (e.g. PROMISE [15]).

In summary, this editorial has presented a very brief (ESTRO-inspired) overview of some exciting research opportunities available in the application of medical image analysis to radiation oncology. This research promises to provide benefits in three main areas: the improvement of radiation treatment dose delivery; a reduction in treatment toxicity and improvements in clinical productivity.