Skip to main content

Evaluation of the Robustness of Learned MR Image Reconstruction to Systematic Deviations Between Training and Test Data for the Models from the fastMRI Challenge

  • Conference paper
  • First Online:
Machine Learning for Medical Image Reconstruction (MLMIR 2021)


The 2019 fastMRI challenge was an open challenge designed to advance research in the field of machine learning for MR image reconstruction. The goal for the participants was to reconstruct undersampled MRI k-space data. The original challenge left an open question as to how well the reconstruction methods will perform in the setting where there is a systematic difference between training and test data. In this work we tested the generalization performance of the submissions with respect to various perturbations, and despite differences in model architecture and training, all of the methods perform very similarly.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 44.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 59.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others


  1. Accelerating cartesian MRI by domain-transform manifold learning in phase-encoding direction. Med. Image Anal. 63, 101689 (2020)

    Google Scholar 

  2. Aggarwal, H.K., Mani, M.P., Jacob, M.: MoDL: model-based deep learning architecture for inverse problems. IEEE Trans. Med. Imaging 38(2), 394–405 (2018)

    Article  Google Scholar 

  3. Eo, T., Jun, Y., Kim, T., Jang, J., Lee, H.J., Hwang, D.: KIKI-net: cross-domain convolutional neural networks for reconstructing undersampled magnetic resonance images. Magn. Reson. Med. 80(5), 2188–2201 (2018)

    Article  Google Scholar 

  4. Hammernik, K., et al.: Learning a variational network for reconstruction of accelerated MRI data. Magn. Reson. Med. 79(6), 3055–3071 (2018)

    Article  Google Scholar 

  5. Hammernik, K., Schlemper, J., Qin, C., Duan, J., Summers, R.M., Rueckert, D.: \(\Sigma \)-Net: ensembled iterative deep neural networks for accelerated parallel MR image reconstruction. In: Proceedings of the International Society of Magnetic Resonance in Medicine (ISMRM), p. 0987 (2020)

    Google Scholar 

  6. Jun, Y., Eo, T., Shin, H., Kim, T., Lee, H.J., Hwang, D.: Parallel imaging in time-of-flight magnetic resonance angiography using deep multistream convolutional neural networks. Magn. Reson. Med. 81(6), 3840–3853 (2019)

    Article  Google Scholar 

  7. Knoll, F., Hammernik, K., Kobler, E., Pock, T., Recht, M.P., Sodickson, D.K.: Assessment of the generalization of learned image reconstruction and the potential for transfer learning. Magn. Reson. Med. 81(1), 116–128 (2019).

    Article  Google Scholar 

  8. Knoll, F., et al.: Deep-learning methods for parallel magnetic resonance imaging reconstruction: a survey of the current approaches, trends, and issues. IEEE Sig. Process. Mag. 37(1), 128–140 (2020).

    Article  Google Scholar 

  9. Knoll, F., et al.: Advancing machine learning for MR image reconstruction with an open competition: overview of the 2019 fastMRI challenge. Magn. Reson. Med. (2020).

  10. Knoll, F., et al.: fastMRI: a publicly available raw k-space and DICOM dataset of knee images for accelerated MR image reconstruction using machine learning. Radiol. Artif. Intell. 2(1), e190007 (2020)

    Article  Google Scholar 

  11. Lønning, K., Putzky, P., Sonke, J.J., Reneman, L., Caan, M.W., Welling, M.: Recurrent inference machines for reconstructing heterogeneous MRI data. Med. Image Anal. 53, 64–78 (2019)

    Article  Google Scholar 

  12. Pezzotti, N., et al.: An adaptive intelligence algorithm for undersampled knee MRI reconstruction. IEEE Access 8, 204825–204838 (2020).

    Article  Google Scholar 

  13. Putzky, P., Welling, M.: Recurrent inference machines for solving inverse problems (2017)

    Google Scholar 

  14. Putzky, P., Welling, M.: Invert to learn to invert. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc. (2019).

  15. Schlemper, J., Caballero, J., Hajnal, J.V., Price, A.N., Rueckert, D.: A deep cascade of convolutional neural networks for dynamic MR image reconstruction. IEEE Trans. Med. Imaging 37(2), 491–503 (2017)

    Article  Google Scholar 

  16. Uecker, M., et al.: ESPIRiT - an eigenvalue approach to autocalibrating parallel MRI: where SENSE meets GRAPPA. Mag. Reson. Med. 71(3), 990–1001 (2014).

    Article  Google Scholar 

  17. Wang, P., Chen, E.Z., Chen, T., Patel, V.M., Sun, S.: Pyramid convolutional RNN for MRI reconstruction. arXiv:1912.00543 (2019)

  18. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  19. Yaman, B., Hosseini, S.A.H., Moeller, S., Ellermann, J., Uğurbil, K., Akçakaya, M.: Self-supervised learning of physics-guided reconstruction neural networks without fully sampled reference data. Magn. Reson. Med. 84(6), 3172–3191 (2020)

    Article  Google Scholar 

  20. Yang, G., et al.: DAGAN: deep de-aliasing generative adversarial networks for fast compressed sensing MRI reconstruction. IEEE Trans. Med. Imaging 37(6), 1310–1321 (2017)

    Article  Google Scholar 

  21. Yang, Y., Sun, J., Li, H., Xu, Z.: Deep ADMM-Net for compressive sensing MRI. In: Advances in Neural Information Processing Systems, pp. 10–18 (2016)

    Google Scholar 

  22. Yu, S., Park, B., Jeong, J.: Deep iterative down-up CNN for image denoising. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (2019)

    Google Scholar 

  23. Zbontar, J., et al.: fastMRI: an open dataset and benchmarks for accelerated MRI. arXiv preprint arXiv:1811.08839 (2018)

  24. Zhao, H., Gallo, O., Frosio, I., Kautz, J.: Loss functions for neural networks for image processing. arXiv (2015)

    Google Scholar 

  25. Zhu, B., Liu, J.Z., Cauley, S.F., Rosen, B.R., Rosen, M.S.: Image reconstruction by domain-transform manifold learning. Nature 555(7697), 487–492 (2018)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Patricia M. Johnson .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Johnson, P.M. et al. (2021). Evaluation of the Robustness of Learned MR Image Reconstruction to Systematic Deviations Between Training and Test Data for the Models from the fastMRI Challenge. In: Haq, N., Johnson, P., Maier, A., Würfl, T., Yoo, J. (eds) Machine Learning for Medical Image Reconstruction. MLMIR 2021. Lecture Notes in Computer Science(), vol 12964. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-88551-9

  • Online ISBN: 978-3-030-88552-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics