Skip to main content

Joint Image and Label Self-super-Resolution

Part of the Lecture Notes in Computer Science book series (LNIP,volume 12965)


We propose a method to jointly super-resolve an anisotropic image volume along with its corresponding voxel labels without external training data. Our method is inspired by internally trained super-resolution, or self-super-resolution (SSR) techniques that target anisotropic, low-resolution (LR) magnetic resonance (MR) images. While resulting images from such methods are quite useful, their corresponding LR labels—derived from either automatic algorithms or human raters—are no longer in correspondence with the super-resolved volume. To address this, we develop an SSR deep network that takes both an anisotropic LR MR image and its corresponding LR labels as input and produces both a super-resolved MR image and its super-resolved labels as output. We evaluated our method with 50 \(T_1\)-weighted brain MR images \(4\times \) down-sampled with 10 automatically generated labels. In comparison to other methods, our method had superior Dice across all labels and competitive metrics on the MR image. Our approach is the first reported method for SSR of paired anisotropic image and label volumes.


  • Super-resolution
  • MRI
  • Segmentation

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD   44.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   59.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions


  1. 1.

    Source code is publicly available at


  1. American College of Radiology: American College of Radiology magnetic resonance imaging accreditation program: Phantom test guidance for use of the large MRI phantom for the ACR MRI accreditation program, p. 16 (2018)

    Google Scholar 

  2. Chaudhari, A.S., et al.: Super-resolution musculoskeletal MRI using deep learning. Magn. Reson. Med. 80(5), 2139–2154 (2018)

    CrossRef  Google Scholar 

  3. Chen, Y., Xie, Y., Zhou, Z., Shi, F., Christodoulou, A.G., Li, D.: Brain MRI super resolution using 3D deep densely connected neural networks. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 739–742 (2018)

    Google Scholar 

  4. Chen, Y., Liu, S., Wang, X.: Learning continuous image representation with local implicit image function. arXiv preprint arXiv:2012.09161 (2020)

  5. Delannoy, Q., et al.: SegSRGAN: super-resolution and segmentation using generative adversarial networks - application to neonatal brain MRI. Computers in Biology and Medicine 120, 103755 (2020)

    CrossRef  Google Scholar 

  6. Delbracio, M., Sapiro, G.: Removing camera shake via weighted Fourier burst accumulation. IEEE Trans. Image Process. 24(11), 3293–3307 (2015)

    CrossRef  MathSciNet  Google Scholar 

  7. Du, J., et al.: Super-resolution reconstruction of single anisotropic 3D MR images using residual convolutional neural network. Neurocomputing 392, 209–220 (2020)

    CrossRef  Google Scholar 

  8. He, Y., et al.: Structured layer surface segmentation for retina OCT using fully convolutional regression networks. Med. Image Anal. 68, 101856 (2021)

    CrossRef  Google Scholar 

  9. Huo, Y., et al.: 3D whole brain segmentation using spatially localized atlas network tiles. Neuroimage 194, 105–119 (2019)

    CrossRef  Google Scholar 

  10. Jog, A., Carass, A., Prince, J.L.: Self super-resolution for magnetic resonance images. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9902, pp. 553–560. Springer, Cham (2016).

    CrossRef  Google Scholar 

  11. LaMontagne, P.J., et al.: OASIS-3: longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and Alzheimer disease. medRxiv (2019)

    Google Scholar 

  12. Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4681–4690 (2017)

    Google Scholar 

  13. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017

    Google Scholar 

  14. Liu, D., Liu, J., Liu, Y., Tao, R., Prince, J.L., Carass, A.: Label super resolution for 3D magnetic resonance images using deformable U-net. In: Medical Imaging 2021: Image Processing, vol. 11596, pp. 597–602. International Society for Optics and Photonics, SPIE (2021)

    Google Scholar 

  15. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2019)

    Google Scholar 

  16. Masutani, E.M., Bahrami, N., Hsiao, A.: Deep learning single-frame and multiframe super-resolution for cardiac MRI. Radiology 295(3), 552–561 (2020)

    CrossRef  Google Scholar 

  17. Pham, C., Ducournau, A., Fablet, R., Rousseau, F.: Brain MRI super-resolution using deep 3D convolutional networks. In: 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), pp. 197–200 (2017)

    Google Scholar 

  18. Pham, D.L., Xu, C., Prince, J.L.: Current methods in medical image segmentation. Ann. Rev. Biomed. Eng. 2(1), 315–337 (2000)

    CrossRef  Google Scholar 

  19. Smith, L.N.: A disciplined approach to neural network hyper-parameters: part 1-learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv:1803.09820 (2018)

  20. Wang, S.L., et al.: Thalamus segmentation using convolutional neural networks. In: Medical Imaging 2021: Image Processing, vol. 11596. International Society for Optics and Photonics, SPIE (2021)

    Google Scholar 

  21. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European Conference on Computer Vision (ECCV), September 2018

    Google Scholar 

  22. Zhao, C., Dewey, B.E., Pham, D.L., Calabresi, P.A., Reich, D.S., Prince, J.L.: SMORE: a self-supervised anti-aliasing and super-resolution algorithm for MRI using deep learning. IEEE Trans. Med. Imaging 40, 805–817 (2020)

    CrossRef  Google Scholar 

  23. Zhou, Z., Rahman Siddiquee, M.M., Tajbakhsh, N., Liang, J.: UNet++: a nested U-net architecture for medical image segmentation. In: Stoyanov, D., et al. (eds.) DLMIA/ML-CDS -2018. LNCS, vol. 11045, pp. 3–11. Springer, Cham (2018).

    CrossRef  Google Scholar 

Download references


This material is supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE-1746891. This work was partially supported by National Multiple Sclerosis Society grant RG-1907-34570, and by the Center for Neuroscience and Regenerative Medicine in the Department of Defense. Theoretical development partially supported by DoD/CDMRP grant MS190131.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Samuel W. Remedios .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Remedios, S.W., Han, S., Dewey, B.E., Pham, D.L., Prince, J.L., Carass, A. (2021). Joint Image and Label Self-super-Resolution. In: Svoboda, D., Burgos, N., Wolterink, J.M., Zhao, C. (eds) Simulation and Synthesis in Medical Imaging. SASHIMI 2021. Lecture Notes in Computer Science(), vol 12965. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-87591-6

  • Online ISBN: 978-3-030-87592-3

  • eBook Packages: Computer ScienceComputer Science (R0)