Temporal Interpolation of Abdominal MRIs Acquired During Free-Breathing

  • Neerav KaraniEmail author
  • Christine Tanner
  • Sebastian Kozerke
  • Ender Konukoglu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10434)


We propose a convolutional neural network (CNN) based solution for temporal image interpolation in navigated 2D multi-slice dynamic MRI acquisitions. Such acquisitions can achieve high contrast time-resolved volumetric images without the need for breath-holding, which makes them essential for quantifying breathing induced motion for MR guided therapies. Reducing the number of navigator slices needed in these acquisitions would allow increasing through-plane resolution and reducing overall acquisition time. The proposed CNN achieves this by interpolating between successive navigator slices. The method is an end-to-end learning based approach and avoids the determination of the motion field between the input images. We evaluate the method on a dataset of abdominal MRI sequences acquired from 14 subjects during free-breathing, which exhibit pseudo-periodic motion and sliding motion interfaces. Compared to an interpolation-by-registration approach, the method achieves higher interpolation accuracy on average, quantified in terms of intensity RMSE and residual motion errors. Further, we analyze the differences between the two methods, showing the CNN’s advantages in peak inhale and exhale positions.



This work was supported by a K40 GPU grant from Nvidia.


  1. 1.
    Von Siebenthal, M., Székely, G., Gamper, U., Boesiger, P., Lomax, A., Cattin, P.: 4D MR imaging of respiratory organ motion and its variability. Phys. Med. Biol. 52, 1547 (2007)CrossRefGoogle Scholar
  2. 2.
    Bert, C., Durante, M.: Motion in radiotherapy: particle therapy. Phys. Med. Biol. 56(16), R113 (2011)CrossRefGoogle Scholar
  3. 3.
    Arnold, P., Preiswerk, F., Fasel, B., Salomir, R., Scheffler, K., Cattin, P.C.: 3D organ motion prediction for MR-guided high intensity focused ultrasound. In: Fichtinger, G., Martel, A., Peters, T. (eds.) MICCAI 2011 Part II. LNCS, vol. 6892, pp. 623–630. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-23629-7_76CrossRefGoogle Scholar
  4. 4.
    Tsao, J., Kozerke, S.: MRI temporal acceleration techniques. J. Magn. Reson. Imaging 36(3), 543 (2012)CrossRefGoogle Scholar
  5. 5.
    Uecker, M., Zhang, S., Voit, D., Karaus, A., Merboldt, K.-D., Frahm, J.: Real-time MRI at a resolution of 20 ms. NMR Biomed. 23(8), 986 (2010)CrossRefGoogle Scholar
  6. 6.
    Tryggestad, E., Flammang, A., Han-Oh, S., Hales, R., Herman, J., McNutt, T., Roland, T., Shea, S.M., Wong, J.: Respiration-based sorting of dynamic MRI to derive representative 4D-MRI for radiotherapy planning. Med. Phys. 40(5), 051909 (2013)CrossRefGoogle Scholar
  7. 7.
    Baumgartner, C.F., Kolbitsch, C., McClelland, J.R., Rueckert, D., King, A.P.: Groupwise simultaneous manifold alignment for high-resolution dynamic MR imaging of respiratory motion. In: Gee, J.C., Joshi, S., Pohl, K.M., Wells, W.M., Zöllei, L. (eds.) IPMI 2013. LNCS, vol. 7917, pp. 232–243. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-38868-2_20CrossRefGoogle Scholar
  8. 8.
    Nam, T.-J., Park, R.-H., Yun, J.-H.: Optical flow based frame interpolation of ultrasound images. In: Campilho, A., Kamel, M.S. (eds.) ICIAR 2006 Part I. LNCS, vol. 4141, pp. 792–803. Springer, Heidelberg (2006). doi: 10.1007/11867586_72CrossRefGoogle Scholar
  9. 9.
    Penney, G.P., Schnabel, J.A., Rueckert, D., Viergever, M.A., Niessen, W.J.: Registration-based interpolation. IEEE Trans. Med. Imaging 23(7), 922 (2004)CrossRefGoogle Scholar
  10. 10.
    Zhang, W., Brady, J.M., Becher, H., Noble, J.A.: Spatio-temporal (2D+T) non-rigid registration of real-time 3D echocardiography and cardiovascular MR image sequences. Phys. Med. Biol. 56(5), 1341 (2011)CrossRefGoogle Scholar
  11. 11.
    Lee, G.-I., Park, R.-H., Song, Y.-S., Kim, C.-A., Hwang, J.-S.: Real-time 3D ultrasound fetal image enhancment techniques using motion-compensated frame rate up-conversion. In: Medical Imaging, p. 375 (2003)Google Scholar
  12. 12.
    Gifani, P., Behnam, H., Haddadi, F., Sani, Z.A., Shojaeifard, M.: Temporal super resolution enhancement of echocardiographic images based on sparse representation. IEEE Trans. Ultrason. Ferroelectr. 63(1), 6 (2016)CrossRefGoogle Scholar
  13. 13.
    Long, G., Kneip, L., Alvarez, J.M., Li, H., Zhang, X., Yu, Q.: Learning image matching by simply watching video. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 434–450. Springer, Cham (2016). doi: 10.1007/978-3-319-46466-4_26CrossRefGoogle Scholar
  14. 14.
    Fischer, P., Dosovitskiy, A., Ilg, E., Häusser, P., Hazırbaş, C., Golkov, V., van der Smagt, P., Cremers, D., Brox, T.: Flownet: learning optical flow with convolutional networks. arXiv:1504.06852 (2015)
  15. 15.
    Srivastava, N., Mansimov, E., Salakhutdinov, R.: Unsupervised learning of video representations using LSTMs. In: ICML, p. 843 (2015)Google Scholar
  16. 16.
    Goroshin, R., Mathieu, M.F., LeCun, Y.: Learning to linearize under uncertainty. In: Advances in Neural Information Processing Systems, p. 1234 (2015)Google Scholar
  17. 17.
    Mathieu, M., Couprie, C., LeCun, Y.: Deep multi-scale video prediction beyond mean square error. arXiv:1511.05440 (2015)
  18. 18.
    Yeh, R., Liu, Z., Goldman, D.B., Agarwala, A.: Semantic facial expression editing using autoencoded flow. arXiv:1611.09961 (2016)
  19. 19.
    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, p. 2672 (2014)Google Scholar
  20. 20.
    Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. arXiv:1312.6114 (2013)
  21. 21.
    Vishnevskiy, V., Gass, T., Szekely, G., Tanner, C., Goksel, O.: Isotropic total variation regularization of displacements in parametric image registration. IEEE Trans. Med. Imaging 36(2), 385 (2016)CrossRefGoogle Scholar
  22. 22.
    Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., et al.: Tensorflow: large-scale machine learning on heterogeneous distributed systems. arXiv:1603.04467 (2016)
  23. 23.
    He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: ICCV, p. 1026 (2015)Google Scholar
  24. 24.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. arXiv:1412.6980 (2014)

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Neerav Karani
    • 1
    Email author
  • Christine Tanner
    • 1
  • Sebastian Kozerke
    • 2
  • Ender Konukoglu
    • 1
  1. 1.Computer Vision LaboratoryETH ZurichZurichSwitzerland
  2. 2.Institute for Biomedical EngineeringUniversity & ETH ZurichZurichSwitzerland

Personalised recommendations