Abstract
Motivated by the recent great success of attention modeling in computer vision, it is highly desired to extend the Transformer architecture from the conventional Euclidean space to non-Euclidean spaces. Given the intrinsic spherical topology of brain cortical surfaces in neuroimaging, in this study, we propose a novel Spherical Transformer, an effective general-purpose backbone using the self-attention mechanism for analysis of cortical surface data represented by triangular meshes. By mapping the cortical surface onto a sphere and splitting it uniformly into overlapping spherical surface patches, we encode the long-range dependency within each patch by the self-attention operation and formulate the cross-patch feature transmission via overlapping regions. By limiting the self-attention computation to local patches, our proposed Spherical Transformer preserves detailed contextual information and enjoys great efficiency with linear computational complexity with respect to the patch size. Moreover, to better process longitudinal cortical surfaces, which are increasingly popular in neuroimaging studies, we unprecedentedly propose the spatiotemporal self-attention operation to jointly extract the spatial context and dynamic developmental patterns within a single layer, thus further enlarging the expressive power of the generated representation. To comprehensively evaluate the performance of our Spherical Transformer, we validate it on a surface-level prediction task and a vertex-level dense prediction task, respectively, i.e., the cognition prediction and cortical thickness map development prediction, which are important in early brain development mapping. Both applications demonstrate the competitive performance of our Spherical Transformer in comparison with the state-of-the-art methods.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., et al.: Transunet: transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306 (2021)
Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., et al.: Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., et al.: An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
Fischl, B.: Freesurfer. Neuroimage 62(2), 774–781 (2012)
Gao, Y., Zhou, M., Metaxas, D.N.: Utnet: a hybrid transformer architecture for medical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 61–71. Springer (2021)
Girault, J.B., et al.: Cortical structure and cognition in infants and toddlers. Cereb. Cortex 30(2), 786–800 (2020)
Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., et al.: A survey on vision transformer. IEEE Trans. Pattern Anal. Mach. Intell. (2022). https://doi.org/10.1109/TPAMI.2022.3152247
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Ji, S., Xu, W., Yang, M., Yu, K.: 3D convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 221–231 (2012)
Ji, Y., Zhang, R., Wang, H., Li, Z., Wu, L., Zhang, S., et al.: Multi-compound transformer for accurate biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 326–336. Springer (2021)
Jiang, C., Huang, J., Kashinath, K., Marcus, P., Niessner, M., et al.: Spherical CNNs on unstructured grids. arXiv preprint arXiv:1901.02039 (2019)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)
LeCun, Y., et al.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4), 541–551 (1989)
Li, G., et al.: Measuring the dynamic longitudinal cortex development in infants by reconstruction of temporally consistent cortical surfaces. Neuroimage 90, 266–279 (2014)
Li, G., Wang, L., Shi, F., Gilmore, J.H., Lin, W., Shen, D.: Construction of 4D high-definition cortical surface atlases of infants: methods and applications. Med. Image Anal. 25(1), 22–36 (2015)
Li, G., et al.: Computational neuroanatomy of baby brains: a review. Neuroimage 185, 906–925 (2019)
Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117–2125 (2017)
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., et al.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021)
Matsoukas, C., Haslum, J.F., Söderberg, M., Smith, K.: Is it time to replace cnns with transformers for medical images? arXiv preprint arXiv:2108.09038 (2021)
Meng, Y., Li, G., Gao, Y., Lin, W., Shen, D.: Learning-based subject-specific estimation of dynamic maps of cortical morphology at missing time points in longitudinal infant studies. Hum. Brain Mapp. 37(11), 4129–4147 (2016)
Mullen, E.M., et al.: Mullen scales of early learning. AGS (ed.). Circle Pines, MN: American Guidance Service Inc. (1995)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241. Springer (2015)
Sun, L., Zhang, D., Lian, C., Wang, L., Wu, Z., Shao, W., et al.: Topological correction of infant white matter surfaces using anatomically constrained convolutional neural network. Neuroimage. 198, 114–124 (2019)
Valanarasu, J.M.J., Oza, P., Hacihaliloglu, I., Patel, V.M.: Medical transformer: gated axial-attention for medical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 36–46. Springer (2021)
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)
Wang, L., Li, G., Shi, F., Cao, X., Lian, C., Nie, D., et al.: Volume-based analysis of 6-month-old infant brain MRI for autism biomarker identification and early diagnosis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 411–419. Springer (2018)
Wu, Z., Wang, L., Lin, W., Gilmore, J.H., Li, G., Shen, D.: Construction of 4D infant cortical surface atlases with sharp folding patterns via spherical patch-based group-wise sparse representation. Hum. Brain Mapp. 40(13), 3860–3880 (2019)
Zhao, F., Wu, Z., Li, G.: Deep learning in cortical surface-based neuroimage analysis: a systematic review. Intell. Med. (2022). https://doi.org/10.1016/j.imed.2022.06.002
Zhao, F., et al.: Spherical deformable U-Net: application to cortical surface parcellation and development prediction. IEEE Trans. Med. Imaging 40(4), 1217–1228 (2021)
Zhao, F., Xia, S., Wu, Z., Duan, D., Wang, L., Lin, W., et al.: Spherical U-Net on cortical surfaces: methods and applications. In: International Conference on Information Processing in Medical Imaging, pp. 855–866. Springer (2019)
Acknowledgements
This work was supported in part by the National Institutes of Health (NIH) under Grants MH116225, MH117943, and MH123202.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Cheng, J. et al. (2022). Spherical Transformer on Cortical Surfaces. In: Lian, C., Cao, X., Rekik, I., Xu, X., Cui, Z. (eds) Machine Learning in Medical Imaging. MLMI 2022. Lecture Notes in Computer Science, vol 13583. Springer, Cham. https://doi.org/10.1007/978-3-031-21014-3_42
Download citation
DOI: https://doi.org/10.1007/978-3-031-21014-3_42
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-21013-6
Online ISBN: 978-3-031-21014-3
eBook Packages: Computer ScienceComputer Science (R0)