Abstract
Multi-modal and multi-contrast imaging datasets have diverse voxel-wise intensities. For example, quantitative MRI acquisition protocols are designed specifically to yield multiple images with widely-varying contrast that inform models relating MR signals to tissue characteristics. The large variance across images in such data prevents the use of standard normalisation techniques, making super resolution highly challenging. We propose a novel self-supervised mixture-of-experts (SS-MoE) paradigm for deep neural networks, and hence present a method enabling improved super resolution of data where image intensities are diverse and have large variance. Unlike the conventional MoE that automatically aggregates expert results for each input, we explicitly assign an input to the corresponding expert based on the predictive pseudo error labels in a self-supervised fashion. A new gater module is trained to discriminate the error levels of inputs estimated by Multiscale Quantile Segmentation. We show that our new paradigm reduces the error and improves the robustness when super resolving combined diffusion-relaxometry MRI data from the Super MUDI dataset. Our approach is suitable for a wide range of quantitative MRI techniques, and multi-contrast or multi-modal imaging techniques in general. It could be applied to super resolve images with inadequate resolution, or reduce the scanning time needed to acquire images of the required resolution. The source code and the trained models are available at https://github.com/hongxiangharry/SS-MoE.
H. Lin and Y. Zhou contributed equally.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The rule empirically selects an equispaced grid along a power-law distribution as class boundaries.
- 2.
Uniformly crop patches by the function extract_patches in scikit-learn 0.22.
References
CDMRI super MUDI challenge 2020. https://www.developingbrain.co.uk/data/
Alexander, D.C., Zikic, D., Ghosh, A., et al.: Image quality transfer and applications in diffusion MRI. Neuroimage 152, 283–298 (2017)
Blumberg, S.B., Tanno, R., Kokkinos, I., Alexander, D.C.: Deeper image quality transfer: training low-memory neural networks for 3D images. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 118–125. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00928-1_14
Chen, G., Dong, B., Zhang, Y., Lin, W., Shen, D., Yap, P.T.: XQ-SR: joint x-q space super-resolution with application to infant diffusion MRI. Med. Image Anal. 57, 44–55 (2019)
Chen, Y., Shi, F., Christodoulou, A.G., Xie, Y., Zhou, Z., Li, D.: Efficient and accurate MRI super-resolution using a generative adversarial network and 3D multi-level densely connected network. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 91–99. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00928-1_11
Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 391–407. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_25
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249–256 (2010)
Gross, S., Ranzato, M., Szlam, A.: Hard mixtures of experts for large scale weakly supervised vision. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5085–5093. IEEE (2017)
Grussu, F., et al.: Multi-parametric quantitative in vivo spinal cord MRI with unified signal readout and image denoising. Neuroimage 217, 116884 (2020)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778. IEEE (2016)
Heinrich, L., Bogovic, J.A., Saalfeld, S.: Deep learning for isotropic super-resolution from non-isotropic 3D electron microscopy. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10434, pp. 135–143. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66185-8_16
Hutter, J., Slator, P.J., Christiaens, D., et al.: Integrated and efficient diffusion-relaxometry using ZEBRA. Sci. Rep. 8(1), 1–13 (2018)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 37, pp. 448–456. PMLR, Lille (2015)
Jacobs, R.A., Jordan, M.I., Nowlan, S.E., Hinton, G.E.: Adaptive mixture of experts (1991)
Jula Vanegas, L., Behr, M., Munk, A.: Multiscale quantile segmentation. J. Am. Stat. Assoc., 1–14 (2021)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Lin, H., et al.: Deep learning for low-field to high-field MR: image quality transfer with probabilistic decimation simulator. In: Knoll, F., Maier, A., Rueckert, D., Ye, J.C. (eds.) MLMIR 2019. LNCS, vol. 11905, pp. 58–70. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-33843-5_6
Lyu, Q., et al.: Multi-contrast super-resolution MRI through a progressive network. IEEE Trans. Med. Imaging 39(9), 2738–2749 (2020)
Ma, J., Yu, J., Liu, S., et al.: PathSRGAN: multi-supervised super-resolution for cytopathological images using generative adversarial network. IEEE Trans. Med. Imaging 39(9), 2920–2930 (2020)
Makkuva, A., Viswanath, P., Kannan, S., Oh, S.: Breaking the gridlock in mixture-of-experts: consistent and efficient algorithms. In: Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 4304–4313. PMLR (2019)
Pizzolato, M., Palombo, M., Bonet-Carne, E., et al.: Acquiring and predicting multidimensional diffusion (MUDI) data: an open challenge. In: Gyori, N., Hutter, J., Nath, V., Palombo, M., Pizzolato, M., Zhang, F. (eds.) Computational Diffusion MRI, pp. 195–208. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-73018-5
Reinhold, J.C., Dewey, B.E., Carass, A., Prince, J.L.: Evaluating the impact of intensity normalization on MR image synthesis. In: Medical Imaging 2019: Image Processing, p. 126. SPIE (2019)
Shen, S., Yao, Z., Gholami, A., Mahoney, M., Keutzer, K.: PowerNorm: rethinking batch normalization in transformers. In: Proceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 119, pp. 8741–8751. PMLR (2020)
Shi, Y., Siddharth, N., Paige, B., Torr, P.: Variational mixture-of-experts autoencoders for multi-modal deep generative models. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R., (eds.) Advances in Neural Information Processing Systems, vol. 32, Curran Associates, Inc., (2019). https://proceedings.neurips.cc/paper/2019/file/0ae775a8cb3b499ad1fca944e6f5c836-Paper.pdf
Tanno, R., Arulkumaran, K., Alexander, D., Criminisi, A., Nori, A.: Adaptive neural trees. In: Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 6166–6175. PMLR (2019)
Tanno, R., et al.: Bayesian image quality transfer with CNNs: exploring uncertainty in dMRI super-resolution. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 611–619. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66182-7_70
Tanno, R., Worrall, D.E., Kaden, E., et al.: Uncertainty modelling in deep learning for safer neuroimage enhancement: demonstration in diffusion MRI. NeuroImage 225, 117366 (2020)
Tong, Q., et al.: Multicenter dataset of multi-shell diffusion MRI in healthy traveling adults with identical settings. Sci. Data 7(1), 1–7 (2020)
Van Essen, D.C., Smith, S.M., Barch, D.M., Behrens, T.E., Yacoub, E., Ugurbil, K.: The WU-MINN human connectome project: an overview. Neuroimage 80, 62–79 (2013)
Van Steenkiste, G., Poot, D.H., Jeurissen, B., et al.: Super-resolution T1 estimation: quantitative high resolution T1 mapping from a set of low resolution T1-weighted images with different slice orientations. Magn. Reson. Med. 77(5), 1818–1830 (2017)
Wilson, A.G., Izmailov, P.: Bayesian deep learning and a probabilistic perspective of generalization. In: Advances in Neural Information Processing Systems, vol. 33, pp. 4697–4708. Curran Associates, Inc. (2020)
Zhang, R., Garrett, J., Ge, Y., Ji, X., Chen, G.H., Li, K.: Design, construction, and initial results of a prototype multi-contrast X-ray breast imaging system. In: Medical Imaging 2018: Physics of Medical Imaging, vol. 176, p. 31. SPIE (2018)
Zhang, Y., Yap, P.T., Chen, G., Lin, W., Wang, L., Shen, D.: Super-resolution reconstruction of neonatal brain magnetic resonance images via residual structured sparse representation. Med. Image Anal. 55, 76–87 (2019)
Zhao, C., Shao, M., Carass, A., et al.: Applications of a deep learning method for anti-aliasing and super-resolution in MRI. Magn. Reson. Imaging 64, 132–141 (2019)
Zheng, Z., et al.: Self-supervised mixture-of-experts by uncertainty estimation. Proc. AAAI Conf. Artif. Intell. 33, 5933–5940 (2019)
Acknowledgements
This work was supported by EPSRC grants EP/M020533/1, EP/R014019/1, and EP/V034537/1 as well as the NIHR UCLH Biomedical Research Centre.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Lin, H., Zhou, Y., Slator, P.J., Alexander, D.C. (2021). Generalised Super Resolution for Quantitative MRI Using Self-supervised Mixture of Experts. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12906. Springer, Cham. https://doi.org/10.1007/978-3-030-87231-1_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-87231-1_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87230-4
Online ISBN: 978-3-030-87231-1
eBook Packages: Computer ScienceComputer Science (R0)
