U-Patch GAN: A Medical Image Fusion Method Based on GAN

Abstract

Although medical imaging is frequently used to diagnose diseases, in complex diagnostic situations, specialists typically need to look at different modalities of image information. Creating a composite multimodal medical image can aid professionals in making quick and accurate diagnoses of diseases. The fused images of many medical image fusion algorithms, however, are frequently unable to precisely retain the functional and structural information of the source image. This work develops an end-to-end model based on GAN (U-Patch GAN) to implement the self-supervised fusion of multimodal brain images in order to enhance the fusion quality. The model uses the classical network U-net as the generator, and it uses the dual adversarial mechanism based on the Markovian discriminator (PatchGAN) to enhance the generator's attention to high-frequency information. To ensure that the network satisfies the Lipschitz continuity, we apply the spectral norm to each layer of the network. We also propose better adversarial loss and feature loss (feature matching loss and VGG-16 perceptual loss) based on the F-norm, which significantly enhance the quality of fused images. On public data sets, we performed a lot of tests. First, we studied how clinically useful the fused image was. The model's performance in single-slice images and continuous-slice images was then confirmed by comparison with other six most popular mainstream fusion approaches. Finally, we verify the effectiveness of the adversarial loss and feature loss.