Generative Adversary Network Based on Cross-Modal Transformer for CT to MR Images Transformation

Abstract

Acquiring Magnetic Resonance (MR) images in the current medical imaging tasks is expensive and time-consuming. We need technology to acquire multi-contrast MR images. Nowadays, studying the synthesis of MR images through deep learning algorithms to improve diagnostic efficiency is a hot topic. However, cross-modal translations are very challenging. This paper proposes an efficient and effective generative adversary network based on Cross-Modal Transformer (C-M Transformer) to address the issues of blurred synthetic images and unstable training, in order to achieve the conversion from Computed Tomography (CT) images to MR images. Firstly, the original input CT image is filtered to generate high-frequency detail images, and then the high-frequency detail images and the original images are respectively fed into the U-shaped network structure for feature extraction. After four downsamplings, the features are sent to the C-M Transformer for feature fusion. In C-M Transformer, the detail feature stream is Q, and the original feature stream is K and V. We added a Masked Attention to provide an average feature representation of two streams. The fused feature image is fed to the upsampling portion of the U-shaped structure to generate the MR image. It can be shown through the experimental results that the method outperforms mainstream algorithms in terms of Mean square error (MAE), Peak signal-to-noise ratio (PSNR), and Structural similarity (SSIM). This method generates MR images that show bone marrow signals in the vertebral body more clearly and accurately than other methods. It can clearly show the position of the lumbar vertebral plate and so on. The results of this method can be used to assist in orthopedic diagnosis after approval by the physician.