Synthesizing Dynamic MRI Using Long-Term Recurrent Convolutional Networks

  • Frank PreiswerkEmail author
  • Cheng-Chieh Cheng
  • Jie Luo
  • Bruno Madore
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11046)


A method is proposed for converting raw ultrasound signals of respiratory organ motion into high frame rate dynamic MRI using a long-term recurrent convolutional neural network. Ultrasound signals were acquired using a single-element transducer, referred to here as ‘organ-configuration motion’ (OCM) sensor, while sagittal MR images were simultaneously acquired. Both streams of data were used for training a cascade of convolutional layers, to extract relevant features from raw ultrasound, followed by a recurrent neural network, to learn its temporal dynamics. The network was trained with MR images on the output, and was employed to predict MR images at a temporal resolution of 100 frames per second, based on ultrasound input alone, without any further MR scanner input. The method was validated on 7 subjects.



Support from grants NIH P41EB015898, R03EB025546, R01CA149342, and R21EB019500 is duly acknowledged. GPU hardware was generously donated by NVIDIA Corporation.


  1. 1.
    Petrusca, L., et al.: Hybrid ultrasound/magnetic resonance simultaneous acquisition and image fusion for motion monitoring in the upper abdomen. Investig. Radiol. 48(5), 333–340 (2013)CrossRefGoogle Scholar
  2. 2.
    Feinberg, D.A., Giese, D., Bongers, D.A., Ramanna, S., Zaitsev, M., Markl, M., Günther, M.: Hybrid ultrasound MRI for improved cardiac imaging and real-time respiration control. Magn. Reson. Med. 63(2), 290–296 (2010)CrossRefGoogle Scholar
  3. 3.
    Preiswerk, F., et al.: Hybrid MRI-ultrasound acquisitions, and scannerless real-time imaging. Magn. Reson. Med. (2016)Google Scholar
  4. 4.
    Donahue, J., et al.: Long-term recurrent convolutional networks for visual recognition and description. CoRR abs/1411.4389 (2014)Google Scholar
  5. 5.
    Schwartz, B.M., McDannold, N.J.: Ultrasound echoes as biometric navigators. Magn. Reson. Med. 69(4), 1023–1033 (2013)CrossRefGoogle Scholar
  6. 6.
    Nadaraya, E.A.: On estimating regression. Theory Probab. & Its Appl. 9(1), 141–142 (1964)CrossRefGoogle Scholar
  7. 7.
    Watson, G.S.: Smooth regression analysis. Sankhyā: Indian J. Stat., Ser. A 26, 359–372 (1964)MathSciNetzbMATHGoogle Scholar
  8. 8.
    Preiswerk, F., et al.: Hybrid Utrasound and MRI Acquisitions for High-Speed Imaging of Respiratory Organ Motion. In: Navab, Nassir, Hornegger, Joachim, Wells, William M., Frangi, Alejandro F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 315–322. Springer, Cham (2015). Scholar
  9. 9.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)CrossRefGoogle Scholar
  10. 10.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  11. 11.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Frank Preiswerk
    • 1
    Email author
  • Cheng-Chieh Cheng
    • 1
  • Jie Luo
    • 1
    • 2
  • Bruno Madore
    • 1
  1. 1.Brigham and Women’s Hospital, Harvard Medical SchoolBostonUSA
  2. 2.Graduate School of Frontier SciencesThe University of TokyoTokyoJapan

Personalised recommendations