Abstract
Considering the difficulty to obtain complete multi-modality MRI scans in some real-world data acquisition situations, synthesizing MRI data is a highly relevant and important topic to complement diagnosis information in clinical practice. In this study, we present a novel MRI synthesizer, called AutoGAN-Synthesizer, which automatically discovers generative networks for cross-modality MRI synthesis. Our AutoGAN-Synthesizer adopts gradient-based search strategies to explore the generator architecture by determining how to fuse multi-resolution features and utilizes GAN-based perceptual searching losses to handle the trade-off between model complexity and performance. Our AutoGAN-Synthesizer can search for a remarkable and light-weight architecture with 6.31 Mb parameters only occupying 12 GPU hours. Moreover, to incorporate richer prior knowledge for MRI synthesis, we derive K-space features containing the low- and high-spatial frequency information and incorporate such features into our model. To our best knowledge, this is the first work to explore AutoML for cross-modality MRI synthesis, and our approach is also capable of tailoring networks given either different multiple modalities or just a single modality as input. Extensive experiments show that our AutoGAN-Synthesizer outperforms the state-of-the-art MRI synthesis methods both quantitatively and qualitatively. The code are available at https://github.com/HUuxiaobin/AutoGAN-Synthesizer.
X. Hu and R. Shen—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Aggarwal, H.K., Mani, M.P., Jacob, M.: Multi-shot sensitivity-encoded diffusion MRI using model-based deep learning (MODL-MUSSELS). In: ISBI, pp. 1541–1544. IEEE (2019)
Armanious, K., et al.: MedGAN: medical image translation using GANs. Comput. Med. Imaging Graph. 79, 101684 (2020)
Bakas, S., et al.: Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci. Data 4, 170117 (2017)
Bakas, S., et al.: Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 (2018)
Chartsias, A., Joyce, T., Giuffrida, M.V., Tsaftaris, S.A.: Multimodal MR synthesis via modality-invariant latent representation. IEEE Trans. Med. Imaging 37(3), 803–814 (2017)
Costa, P., et al.: End-to-end adversarial retinal image synthesis. IEEE Trans. Med. Imaging 37(3), 781–791 (2017)
Dar, S.U., Yurt, M., Karacan, L., Erdem, A., Erdem, E., Çukur, T.: Image synthesis in multi-contrast MRI with conditional generative adversarial networks. IEEE Trans. Med. Imaging 38(10), 2375–2388 (2019)
Gou, Y., Li, B., Liu, Z., Yang, S., Peng, X.: Clearer: multi-scale neural architecture search for image restoration. NeurIPS. 33, 17129–17140 (2020)
Han, Y., Sunwoo, L., Ye, J.C.: k-space deep learning for accelerated MRI. IEEE Trans. Med. Imaging 39(2), 377–386 (2019)
Hu, X.: Multi-texture GAN: exploring the multi-scale texture translation for brain MR images. arXiv preprint arXiv:2102.07225 (2021)
Huang, Y., Shao, L., Frangi, A.F.: Simultaneous super-resolution and cross-modality synthesis of 3d medical images using weakly-supervised joint convolutional sparse coding. In: CVPR, pp. 6070–6079 (2017)
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR, pp. 1125–1134 (2017)
Joyce, T., Chartsias, A., Tsaftaris, S.A.: Robust multi-modal MR image synthesis. In: Descoteaux, M., et al. (eds.) MICCAI 2017. LNCS, vol. 10435, pp. 347–355. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66179-7_40
Jun, Y., Shin, H., Eo, T., Hwang, D.: Joint deep model-based MR image and coil sensitivity reconstruction network (Joint-ICNet) for fast MRI. In: CVPR, pp. 5270–5279 (2021)
Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: International Conference on Learning Representations (2018)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Liu, C., et al.: Progressive neural architecture search. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11205, pp. 19–35. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01246-5_2
Liu, H., Simonyan, K., Yang, Y.: Darts: Differentiable architecture search. In: ICLR (2018)
Liu, Z., Wang, H., Zhang, S., Wang, G., Qi, J.: NAS-SCAM: neural architecture search-based spatial and channel joint attention module for nuclei semantic segmentation and classification. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12261, pp. 263–272. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59710-8_26
Loshchilov, I., Hutter, F.: SGDR: stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (2016)
Menze, B.H., et al.: The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans. Med. Imaging 34(10), 1993–2024 (2014)
Nah, S., Hyun Kim, T., Mu Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: CVPR, pp. 3883–3891 (2017)
Nie, D., et al.: Medical image synthesis with deep convolutional adversarial networks. IEEE Trans. Biomed. Eng. 65(12), 2720–2730 (2018)
Qianye, Y., Li, N., Zhao, Z., Xingyu, F., Eric, I., Chang, C., Xu, Y.: MRI cross-modality image-to-image translation. Sci. Rep. 10(1), 1–8 (2020)
Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. In: AAAI, vol. 33, pp. 4780–4789 (2019)
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Sharma, A., Hamarneh, G.: Missing MRI pulse sequence synthesis using multi-modal generative adversarial network. IEEE Trans. Med. Imaging 39(4), 1170–1183 (2019)
Wang, J., et al.: Deep high-resolution representation learning for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 43, 3349–3364 (2020)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Welander, P., Karlsson, S., Eklund, A.: Generative adversarial networks for image-to-image translation on multi-contrast MR images-a comparison of cyclegan and unit. arXiv preprint arXiv:1806.07777 (2018)
Yan, X., Jiang, W., Shi, Y., Zhuo, C.: MS-NAS: multi-scale neural architecture search for medical image segmentation. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12261, pp. 388–397. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59710-8_38
Yu, B., Zhou, L., Wang, L., Shi, Y., Fripp, J., Bourgeat, P.: EA-GANs: edge-aware generative adversarial networks for cross-modality MR image synthesis. IEEE Trans. Med. Imaging 38(7), 1750–1762 (2019)
Yu, B., Zhou, L., Wang, L., Shi, Y., Fripp, J., Bourgeat, P.: Sample-adaptive GANs: linking global and local mappings for cross-modality MR image synthesis. IEEE Trans. Med. Imaging 39(7), 2339–2350 (2020)
Yurt, M., Dar, S.U., Erdem, A., Erdem, E., Oguz, K.K., Çukur, T.: mustGAN: multi-stream generative adversarial networks for MR image synthesis. Med. Image Anal. 70, 101944 (2021)
Zhang, H., Li, Y., Chen, H., Shen, C.: Memory-efficient hierarchical neural architecture search for image denoising. In: CVPR, pp. 3657–3666 (2020)
Zhang, R., Pfister, T., Li, J.: Harmonic unpaired image-to-image translation. arXiv preprint arXiv:1902.09727 (2019)
Zhang, Y., Li, K., Li, K., Fu, Y.: MR image super-resolution with squeeze and excitation reasoning attention network. In: CVPR, pp. 13425–13434 (2021)
Zhou, T., Fu, H., Chen, G., Shen, J., Shao, L.: Hi-net: hybrid-fusion network for multi-modal MR image synthesis. IEEE Trans. Med. Imaging 39(9), 2772–2781 (2020)
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: ICCV, pp. 2223–2232 (2017)
Zhu, Z., Liu, C., Yang, D., Yuille, A., Xu, D.: V-NAS: neural architecture search for volumetric medical image segmentation. In: 3DV, pp. 240–248. IEEE (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hu, X., Shen, R., Luo, D., Tai, Y., Wang, C., Menze, B.H. (2022). AutoGAN-Synthesizer: Neural Architecture Search for Cross-Modality MRI Synthesis. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13436. Springer, Cham. https://doi.org/10.1007/978-3-031-16446-0_38
Download citation
DOI: https://doi.org/10.1007/978-3-031-16446-0_38
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16445-3
Online ISBN: 978-3-031-16446-0
eBook Packages: Computer ScienceComputer Science (R0)
