Abstract
Homologous anatomical landmarks between medical scans are instrumental in quantitative assessment of image registration quality in various clinical applications, such as MRI-ultrasound registration for tissue shift correction in ultrasound-guided brain tumor resection. While manually identified landmark pairs between MRI and ultrasound (US) have greatly facilitated the validation of different registration algorithms for the task, the procedure requires significant expertise, labor, and time, and can be prone to inter- and intra-rater inconsistency. So far, many traditional and machine learning approaches have been presented for anatomical landmark detection, but they primarily focus on mono-modal applications. Unfortunately, despite the clinical needs, inter-modal/contrast landmark detection has very rarely been attempted. Therefore, we propose a novel contrastive learning framework to detect corresponding landmarks between MRI and intra-operative US scans in neurosurgery. Specifically, two convolutional neural networks were trained jointly to encode image features in MRI and US scans to help match the US image patch that contain the corresponding landmarks in the MRI. We developed and validated the technique using the public RESECT database. With a mean landmark detection accuracy of 5.88±4.79 mm against 18.78±4.77 mm with SIFT features, the proposed method offers promising results for MRI-US landmark detection in neurosurgical applications for the first time.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Holland, E.C.: Progenitor cells and glioma formation. Curr. Opin. Neurol. 14(6), 683–688 (2001)
Dolecek, T.A., Propp, J.M., Stroup, N.E., Kruchko, C.: CBTRUS statistical report: primary brain and central nervous system tumors diagnosed in the united states in 2005–2009. Neuro-oncology 14(suppl_5), v1–v49 (2012)
Xiao, Y., et al.: Evaluation of MRI to ultrasound registration methods for brain shift correction: the CuRIOUS2018 challenge. IEEE Trans. Med. Imaging 39(3), 777–786 (2019)
Yao, Q., Xiao, L., Liu, P., Zhou, S.K.: Label-free segmentation of COVID-19 lesions in lung CT. IEEE Trans. Med. Imaging 40(10), 2808–2819 (2021)
Ghesu, F.C., Georgescu, B., Mansi, T., Neumann, D., Hornegger, J., Comaniciu, D.: An artificial agent for anatomical landmark detection in medical images. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9902, pp. 229–237. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46726-9_27
Zhu, H., Yao, Q., Xiao, L., Zhou, S.K.: You only learn once: universal anatomical landmark detection. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12905, pp. 85–95. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87240-3_9
Tripathi, A., et al.: Unsupervised landmark detection and classification of lung infection using transporter neural networks. Comput. Biol. Med. 152, 106345 (2023)
Toews, M., Wells, W.M., III.: Efficient and robust model-to-image alignment using 3d scale-invariant features. Med. Image Anal. 17(3), 271–282 (2013)
Salari, S., Rasoulian, A., Battie, M., Fortin, M., Rivaz, H., Xiao, Y.: Uncertainty-aware transformer model for anatomical landmark detection in paraspinal muscle MRIs. In: Medical Imaging,: Image Processing, vol. 12464, pp. 238–244. SPIE (2023)
Xiao, Y., Fortin, M., Unsgård, G., Rivaz, H., Reinertsen, I.: Re trospective evaluation of cerebral tumors (RESECT): a clinical database of pre-operative MRI and intra-operative ultrasound in low-grade glioma surgeries. Med. Phys. 44(7), 3875–3882 (2017)
Rister, B., Horowitz, M.A., Rubin, D.L.: Volumetric image registration from invariant keypoints. IEEE Trans. Image Process. 26(10), 4900–4910 (2017)
You, K., Lee, S., Jo, K., Park, E., Kooi, T., Nam, H.: Intra-class contrastive learning improves computer aided diagnosis of breast cancer in mammography. In: Medical Image Computing and Computer Assisted Intervention-MICCAI: 25th International Conference, Singapore, 18–22 September 2022, Proceedings, Part III, pp. 55–64. Springer (2022). https://doi.org/10.1007/978-3-031-16437-8_6
Cheng, L.-H., Sun, X., van der Geest, R.J.: Contrastive learning for echocardiographic view integration. In: Medical Image Computing and Computer Assisted Intervention-MICCAI,: 25th International Conference, Singapore, v September 2022, Proceedings, Part IV, pp. 340–349. Springer (2022). https://doi.org/10.1007/978-3-031-16440-8_33
Bhattacharya, D., et al.: Supervised contrastive learning to classify paranasal anomalies in the maxillary sinus. In: Medical Image Computing and Computer Assisted Intervention-MICCAI, et al.: 25th International Conference, Singapore, 18–22 September 2022, Proceedings, Part III, pp. 429–438. Springer (2022). https://doi.org/10.1007/978-3-031-16437-8_41
Emre, T., Chakravarty, A., Rivail, A., Riedl, S., Schmidt-Erfurth, U., Bogunović, H.: Tinc: temporally informed non-contrastive learning for disease progression modeling in retinal OCT volumes. In: Medical Image Computing and Computer Assisted Intervention-MICCAI,: 25th International Conference, Singapore, 18–22 September 2022, Proceedings, Part II, pp. 625–634. Springer (2022). https://doi.org/10.1007/978-3-031-16434-7_60
Liu, T., Liu, W., Yu, L., Wan, L., Han, T., Zhu, L.: Joint prediction of meningioma grade and brain invasion via task-aware contrastive learning. In: Medical Image Computing and Computer Assisted Intervention-MICCAI,: 25th International Conference, Singapore, 18–22 September 2022, Proceedings, Part III, pp. 355–365. Springer (2022). https://doi.org/10.1007/978-3-031-16437-8_34
Pan, Y., Gernand, A.D., Goldstein, J.A., Mithal, L., Mwinyelle, D., Wang, J.Z.: Vision-language contrastive learning approach to robust automatic placenta analysis using photographic images. In: Medical Image Computing and Computer Assisted Intervention-MICCAI: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part III, pp. 707–716. Springer (2022). https://doi.org/10.1007/978-3-031-16437-8_68
Hang, W., Huang, Y., Liang, S., Lei, B., Choi, K.-S., Qin, J.: Reliability-aware contrastive self-ensembling for semi-supervised medical image classification. In: Medical Image Computing and Computer Assisted Intervention-MICCAI,: 25th International Conference, Singapore, 18–22 September 2022, Proceedings, Part I, pp. 754–763. Springer (2022). https://doi.org/10.1007/978-3-031-16431-6_71
Quan, Q., Yao, Q., Li, J., Zhou, S.K.: Which images to label for few-shot medical landmark detection?. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20606–20616 (2022)
Quan, Q., Yao, Q., Li, J., et al.: Information-guided pixel augmentation for pixel-wise contrastive learning, arXiv preprint arXiv:2211.07118 (2022)
Yao, Q., Quan, Q., Xiao, L., Kevin Zhou, S.: One-shot medical landmark detection. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12902, pp. 177–188. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87196-3_17
Gutmann, M., Hyvärinen, A.: Noise-contrastive estimation: a new estimation principle for unnormalized statistical models. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, pp. 297–304 (2010)
Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.03748 (2018)
Hering, A., et al.: Learn2reg: comprehensive multi-task medical image registration challenge, dataset and evaluation in the era of deep learning. IEEE Trans. Med. Imaging (2022)
Pirhadi, A., Salari, S., Ahmad, M.O., Rivaz, H., Xiao, Y.: Robust landmark-based brain shift correction with a Siamese neural network in ultrasound-guided brain tumor resection. Inter. J. Comput. Assisted Radiol. Surgery, 1–8 (2022)
Luo, J., et al.: A feature-driven active framework for ultrasound-based brain shift compensation. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11073, pp. 30–38. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00937-3_4
Canalini, L., Klein, J., Miller, D., Kikinis, R.: Segmentation-based registration of ultrasound volumes for glioma resection in image-guided neurosurgery. Int. J. Comput. Assist. Radiol. Surg. 14, 1697–1713 (2019)
Acknowledgment
We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) and Fonds de Recherche du Québec Nature et technologies (FRQNT).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Salari, S., Rasoulian, A., Rivaz, H., Xiao, Y. (2023). Towards Multi-modal Anatomical Landmark Detection for Ultrasound-Guided Brain Tumor Resection with Contrastive Learning. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14228. Springer, Cham. https://doi.org/10.1007/978-3-031-43996-4_64
Download citation
DOI: https://doi.org/10.1007/978-3-031-43996-4_64
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43995-7
Online ISBN: 978-3-031-43996-4
eBook Packages: Computer ScienceComputer Science (R0)
