Advertisement

Robust Multi-modal MR Image Synthesis

  • Thomas JoyceEmail author
  • Agisilaos Chartsias
  • Sotirios A. Tsaftaris
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10435)

Abstract

We present a multi-input encoder-decoder neural network model able to perform MR image synthesis from any subset of its inputs, outperforming prior methods in both single and multi-input settings. This is achieved by encouraging the network to learn a modality invariant latent embedding during training. We demonstrate that a spatial transformer module [7] can be included in our model to automatically correct misalignment in the input data. Thus, our model is robust both to missing and misaligned data at test time. Finally, we show that the model’s modular nature allows transfer learning to different datasets.

Keywords

MRI Synthesis Neural network Brain 

Notes

Acknowledgements

This work was supported in part by the US National Institutes of Health (2R01HL091989-05) and UK EPSRC (EP/P022928/1). We thank NVIDIA for donating a Titan X GPU.

References

  1. 1.
    Bowles, C., et al.: Pseudo-healthy image synthesis for white matter lesion segmentation. In: Tsaftaris, S.A., Gooya, A., Frangi, A.F., Prince, J.L. (eds.) SASHIMI 2016. LNCS, vol. 9968, pp. 87–96. Springer, Cham (2016). doi: 10.1007/978-3-319-46630-9_9 CrossRefGoogle Scholar
  2. 2.
    Burgos, N., Cardoso, M.J., Thielemans, K., Modat, M., Pedemonte, S., Dickson, J., Barnes, A., Ahmed, R., Mahoney, C.J., Schott, J.M.: Attenuation correction synthesis for hybrid PET-MR scanners: application to brain studies. IEEE TMI 33(12), 2332–2341 (2014)Google Scholar
  3. 3.
    Drozdzal, M., Vorontsov, E., Chartrand, G., Kadoury, S., Pal, C.: The importance of skip connections in biomedical image segmentation. In: Carneiro, G., et al. (eds.) LABELS/DLMIA -2016. LNCS, vol. 10008, pp. 179–187. Springer, Cham (2016). doi: 10.1007/978-3-319-46976-8_19 CrossRefGoogle Scholar
  4. 4.
    Huang, Y., Beltrachini, L., Shao, L., Frangi, A.F.: Geometry regularized joint dictionary learning for cross-modality image synthesis in magnetic resonance imaging. In: Tsaftaris, S.A., Gooya, A., Frangi, A.F., Prince, J.L. (eds.) SASHIMI 2016. LNCS, vol. 9968, pp. 118–126. Springer, Cham (2016). doi: 10.1007/978-3-319-46630-9_12 CrossRefGoogle Scholar
  5. 5.
    Huynh, T., Gao, Y., Kang, J., Wang, L., Zhang, P., Lian, J., Shen, D.: Estimating CT image from MRI data using structured random forest and auto-context model. IEEE TMI 35(1), 174–183 (2016)Google Scholar
  6. 6.
    Iglesias, J.E., Konukoglu, E., Zikic, D., Glocker, B., Leemput, K., Fischl, B.: Is synthesizing MRI contrast useful for inter-modality analysis? In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013. LNCS, vol. 8149, pp. 631–638. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-40811-3_79 CrossRefGoogle Scholar
  7. 7.
    Jaderberg, M., Simonyan, K., Zisserman, A.: Spatial transformer networks. In: NIPS, pp. 2017–2025 (2015)Google Scholar
  8. 8.
    Jog, A., Carass, A., Roy, S., Pham, D.L., Prince, J.L.: MR image synthesis by contrast learning on neighborhood ensembles. MIA 24(1), 63–76 (2015)Google Scholar
  9. 9.
    Jog, A., Carass, A., Roy, S., Pham, D.L., Prince, J.L.: Random forest regression for magnetic resonance image synthesis. MIA 35, 475–488 (2017)Google Scholar
  10. 10.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint (2014). arXiv:1412.6980
  11. 11.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). doi: 10.1007/978-3-319-24574-4_28 CrossRefGoogle Scholar
  12. 12.
    Roy, S., Chou, Y.-Y., Jog, A., Butman, J.A., Pham, D.L.: Patch based synthesis of whole head MR images: application To EPI distortion correction. In: Tsaftaris, S.A., Gooya, A., Frangi, A.F., Prince, J.L. (eds.) SASHIMI 2016. LNCS, vol. 9968, pp. 146–156. Springer, Cham (2016). doi: 10.1007/978-3-319-46630-9_15 CrossRefGoogle Scholar
  13. 13.
    Roy, S., Wang, W.-T., Carass, A., Prince, J.L., Butman, J.A., Pham, D.L.: PET attenuation correction using synthetic CT from ultrashort echo-time MR imaging. J. Nuclear Med. 55(12), 2071–2077 (2014)CrossRefGoogle Scholar
  14. 14.
    Roy, S., Carass, A., Prince, J.L.: Magnetic resonance image example-based contrast synthesis. IEEE TMI 32(12), 2348–2363 (2013)Google Scholar
  15. 15.
    Sevetlidis, V., Giuffrida, M.V., Tsaftaris, S.A.: Whole image synthesis using a deep encoder-decoder network. In: Tsaftaris, S.A., Gooya, A., Frangi, A.F., Prince, J.L. (eds.) SASHIMI 2016. LNCS, vol. 9968, pp. 127–137. Springer, Cham (2016). doi: 10.1007/978-3-319-46630-9_13 CrossRefGoogle Scholar
  16. 16.
    van Tulder, G., Bruijne, M.: Why does synthesized data improve multi-sequence classification? In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 531–538. Springer, Cham (2015). doi: 10.1007/978-3-319-24553-9_65 CrossRefGoogle Scholar
  17. 17.
    Van Nguyen, H., Zhou, K., Vemulapalli, R.: Cross-domain synthesis of medical images using efficient location-sensitive deep network. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 677–684. Springer, Cham (2015). doi: 10.1007/978-3-319-24553-9_83 CrossRefGoogle Scholar
  18. 18.
    Ye, D.H., Zikic, D., Glocker, B., Criminisi, A., Konukoglu, E.: Modality propagation: coherent synthesis of subject-specific scans with data-driven regularization. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013. LNCS, vol. 8149, pp. 606–613. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-40811-3_76 CrossRefGoogle Scholar
  19. 19.
    Yosinski, J., Clune, J., Bengio, Y., Lipson, H.: How transferable are features in deep neural networks? In: NIPS, pp. 3320–3328 (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Thomas Joyce
    • 1
    Email author
  • Agisilaos Chartsias
    • 1
  • Sotirios A. Tsaftaris
    • 1
  1. 1.Institute for Digital Communications, School of EngineeringUniversity of EdinburghEdinburghUK

Personalised recommendations