Abstract
This paper demonstrates the potential for synthesis of medical images in one modality (e.g. MR) from images in another (e.g. CT) using a CycleGAN [24] architecture. The synthesis can be learned from unpaired images, and applied directly to expand the quantity of available training data for a given task. We demonstrate the application of this approach in synthesising cardiac MR images from CT images, using a dataset of MR and CT images coming from different patients. Since there can be no direct evaluation of the synthetic images, as no ground truth images exist, we demonstrate their utility by leveraging our synthetic data to achieve improved results in segmentation. Specifically, we show that training on both real and synthetic data increases accuracy by 15% compared to real data. Additionally, our synthetic data is of sufficient quality to be used alone to train a segmentation neural network, that achieves 95% of the accuracy of the same model trained on real data.
A. Chartsias and T. Joyce—Contributed equally.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
References
Alessandrini, M., De Craene, M., Bernard, O., Giffard-Roisin, S., Allain, P., Waechter-Stehle, I., Weese, J., Saloux, E., Delingette, H., Sermesant, M.: A pipeline for the generation of realistic 3D synthetic echocardiographic sequences: methodology and open-access database. IEEE TMI 34(7), 1436–1451 (2015)
Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein gan. preprint arXiv:1701.07875 (2017)
Berthelot, D., Schumm, T., Metz, L.: BEGAN: Boundary Equilibrium Generative Adversarial Networks. preprint arXiv:1703.10717 (2017)
Cordier, N., Delingette, H., Lê, M., Ayache, N.: Extended modality propagation: image synthesis of pathological cases. IEEE TMI 35(12), 2598–2608 (2016)
Duchateau, N., Sermesant, M., Delingette, H., Ayache, N.: Model-based generation of large databases of cardiac images: synthesis of pathological cine MR sequences from real healthy cases. IEEE TMI (99) (2017). doi:10.1109/TMI.2017.2714343
Goodfellow, I., et al.: Generative adversarial nets. In: NIPS, pp. 2672–2680 (2014)
Huang, Y., Beltrachini, L., Shao, L., Frangi, A.F.: Geometry regularized joint dictionary learning for cross-modality image synthesis in magnetic resonance imaging. In: Tsaftaris, S.A., Gooya, A., Frangi, A.F., Prince, J.L. (eds.) SASHIMI 2016. LNCS, vol. 9968, pp. 118–126. Springer, Cham (2016). doi:10.1007/978-3-319-46630-9_12
Huang, Y., Shao, L., Frangi, A.F.: Simultaneous super-resolution and cross-modality synthesis of 3D medical images using weakly-supervised joint convolutional sparse coding. preprint arXiv:1705.02596 (2017)
Iglesias, J.E., Konukoglu, E., Zikic, D., Glocker, B., Van Leemput, K., Fischl, B.: Is synthesizing MRI contrast useful for inter-modality analysis? In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013. LNCS, vol. 8149, pp. 631–638. Springer, Heidelberg (2013). doi:10.1007/978-3-642-40811-3_79
Jog, A., Carass, A., Roy, S., Pham, D.L., Prince, J.L.: Random forest regression for magnetic resonance image synthesis. Med. Image Anal. 35, 475–488 (2017)
Kingma, D., Ba, J.: Adam: a method for stochastic optimization. preprint arXiv:1412.6980 (2014)
Oktay, O., et al.: Multi-input cardiac image super-resolution using convolutional neural networks. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9902, pp. 246–254. Springer, Cham (2016). doi:10.1007/978-3-319-46726-9_29
Oktay, O., Ferrante, E., Kamnitsas, K., Heinrich, M., Bai, W., Caballero, J., Guerrero, R., Cook, S., de Marvao, A., O’Regan, D.: Anatomically constrained neural networks (ACNN): application to cardiac image enhancement and segmentation. preprint arXiv:1705.08302 (2017)
Prakosa, A., Sermesant, M., Delingette, H., Marchesseau, S., Saloux, E., Allain, P., Villain, N., Ayache, N.: Generation of synthetic but visually realistic time series of cardiac images combining a biophysical model and clinical images. IEEE TMI 32(1), 99–109 (2013)
Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. preprint arXiv:1511.06434 (2015)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). doi:10.1007/978-3-319-24574-4_28
Sevetlidis, V., Giuffrida, M.V., Tsaftaris, S.A.: Whole image synthesis using a deep encoder-decoder network. In: Tsaftaris, S.A., Gooya, A., Frangi, A.F., Prince, J.L. (eds.) SASHIMI 2016. LNCS, vol. 9968, pp. 127–137. Springer, Cham (2016). doi:10.1007/978-3-319-46630-9_13
Tavakoli, V., Amini, A.A.: A survey of shaped-based registration and segmentation techniques for cardiac images. Comput. Vis. Image Underst. 117(9), 966–989 (2013)
Tran, P.V.: A fully convolutional neural network for cardiac segmentation in shortaxis MRI. preprint arXiv:1604.00494 (2016)
van Tulder, G., de Bruijne, M.: Why does synthesized data improve multi-sequence classification? In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 531–538. Springer, Cham (2015). doi:10.1007/978-3-319-24553-9_65
Van Nguyen, H., Zhou, K., Vemulapalli, R.: Cross-domain synthesis of medical images using efficient location-sensitive deep network. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 677–684. Springer, Cham (2015). doi:10.1007/978-3-319-24553-9_83
Vemulapalli, R., Van Nguyen, H., Kevin Zhou, S.: Unsupervised cross-modal synthesis of subject-specific scans. In: IEEE ICCV, pp. 630–638 (2015)
Zhou, Y., Giffard-Roisin, S., De Craene, M., D’hooge, J., Alessandrini, M., Friboulet, D., Sermesant, M., Bernard, O.: A framework for the generation of realistic synthetic cardiac ultrasound and magnetic resonance imaging sequences from the same virtual patients. IEEE TMI (99) (2017). doi:10.1109/TMI.2017.2708159
Zhu, J.: Unpaired image-to-image translation using cycle-consistent adversarial networks. preprint arXiv:1703.10593 (2017)
Zhuang, X., Rhode, K.S., Razavi, R.S., Hawkes, D.J., Ourselin, S.: A registrationbased propagation framework for automatic whole heart segmentation of cardiac MRI. IEEE TMI 29(9), 1612–1625 (2010)
Zhuang, X., Shen, J.: Multi-scale patch and multi-modality atlases for whole heart segmentation of MRI. Med. Image Anal. 31, 77–87 (2016)
Acknowledgements
This work was supported in part by the US National Institutes of Health (2R01HL091989-05) and UK EPSRC (EP/P022928/1). We thank NVIDIA for donating a Titan X GPU.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Chartsias, A., Joyce, T., Dharmakumar, R., Tsaftaris, S.A. (2017). Adversarial Image Synthesis for Unpaired Multi-modal Cardiac Data. In: Tsaftaris, S., Gooya, A., Frangi, A., Prince, J. (eds) Simulation and Synthesis in Medical Imaging. SASHIMI 2017. Lecture Notes in Computer Science(), vol 10557. Springer, Cham. https://doi.org/10.1007/978-3-319-68127-6_1
Download citation
DOI: https://doi.org/10.1007/978-3-319-68127-6_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-68126-9
Online ISBN: 978-3-319-68127-6
eBook Packages: Computer ScienceComputer Science (R0)