Abstract
We present a novel method for intraoperative patient-to-image registration by learning Expected Appearances. Our method uses preoperative imaging to synthesize patient-specific expected views through a surgical microscope for a predicted range of transformations. Our method estimates the camera pose by minimizing the dissimilarity between the intraoperative 2D view through the optical microscope and the synthesized expected texture. In contrast to conventional methods, our approach transfers the processing tasks to the preoperative stage, reducing thereby the impact of low-resolution, distorted, and noisy intraoperative images, that often degrade the registration accuracy. We applied our method in the context of neuronavigation during brain surgery. We evaluated our approach on synthetic data and on retrospective data from 6 clinical cases. Our method outperformed state-of-the-art methods and achieved accuracies that met current clinical standards.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ulyanov, D., et al.: Texture networks: feed-forward synthesis of textures and stylized images. In: Proceedings of Machine Learning Research, vol. 48, pp. 1349–1357. PMLR (2016)
Maier-Hein, L., et al.: Surgical data science - from concepts toward clinical translation. Med. Image Anal. 76, 102–306 (2022)
Fernandez, V., et al.: Can segmentation models be trained with fully synthetically generated data? In: Zhao, C., Svoboda, D., Wolterink, J.M., Escobar, M. (eds.) SASHIMI 2022. LNCS, vol. 13570, pp. 79–90. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16980-9_8
Frisken, S., et al.: A comparison of thin-plate spline deformation and finite element modeling to compensate for brain shift during tumor resection. Int. J. Comput. Assist. Radiol. Surg. 15, 75–85 (2019). https://doi.org/10.1007/s11548-019-02057-2
González-Darder, J.M.: ‘State of the art’ of the craniotomy in the early twenty-first century and future development. In: González-Darder, J.M. (ed.) Trepanation, Trephining and Craniotomy, pp. 421–427. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22212-3_34
Haouchine, N., Juvekar, P., Wells III, W.M., Cotin, S., Golby, A., Frisken, S.: Deformation aware augmented reality for craniotomy using 3D/2D non-rigid registration of cortical vessels. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12264, pp. 735–744. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59719-1_71
Haouchine, N., Juvekar, P., Nercessian, M., Wells III, W.M., Golby, A., Frisken, S.: Pose estimation and non-rigid registration for augmented reality during neurosurgery. IEEE Trans. Biomed. Eng. 69(4), 1310–1317 (2022)
Ji, S., Fan, X., Roberts, D.W., Hartov, A., Paulsen, K.D.: Cortical surface shift estimation using stereovision and optical flow motion tracking via projection image registration. Med. Image Anal. 18(7), 1169–1183 (2014)
Ji, S., Wu, Z., Hartov, A., Roberts, D.W., Paulsen, K.D.: Mutual-information-based image to patient re-registration using intraoperative ultrasound in image-guided neurosurgery. Med. Phys. 35(10), 4612–4624 (2008)
Jiang, J., et al.: Marker-less tracking of brain surface deformations by non-rigid registration integrating surface and vessel/sulci features. Int. J. Comput. Assist. Radiol. Surg. 11, 1687–1701 (2016). https://doi.org/10.1007/s11548-016-1358-7
Kuhnt, D., Bauer, M.H.A., Nimsky, C.: Brain shift compensation and neurosurgical image fusion using intraoperative MRI: current status and future challenges. Crit. Rev. Trade Biomed. Eng. 40(3), 175–185 (2012)
Lecomte, F., Dillenseger, J.L., Cotin, S.: CNN-based real-time 2D–3D deformable registration from a single X-ray projection. CoRR abs/2003.08934 (2022)
Luo, M., Larson, P.S., Martin, A.J., Konrad, P.E., Miga, M.I.: An integrated multi-physics finite element modeling framework for deep brain stimulation: preliminary study on impact of brain shift on neuronal pathways. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11768, pp. 682–690. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32254-0_76
Machado, I., et al.: Non-rigid registration of 3d ultrasound for neurosurgery using automatic feature detection and matching. Int. J. Comput. Assist. Radiol. Surg. 13, 1525–1538 (2018). https://doi.org/10.1007/s11548-018-1786-7
Marreiros, F.M.M., Rossitti, S., Wang, C., Smedby, Ö.: Non-rigid deformation pipeline for compensation of superficial brain shift. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013. LNCS, vol. 8150, pp. 141–148. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40763-5_18
Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. Commun. ACM 65(1), 99–106 (2021)
Mohammadi, A., Ahmadian, A., Azar, A.D., Sheykh, A.D., Amiri, F., Alirezaie, J.: Estimation of intraoperative brain shift by combination of stereovision and doppler ultrasound: phantom and animal model study. Int. J. Comput. Assist. Radiol. Surg. 10(11), 1753–1764 (2015). https://doi.org/10.1007/s11548-015-1216-z
Nercessian, M., Haouchine, N., Juvekar, P., Frisken, S., Golby, A.: Deep cortical vessel segmentation driven by data augmentation with neural image analogy. In: 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 721–724. IEEE (2021)
Pereira, V.M., et al.: Volumetric measurements of brain shift using intraoperative cone-beam computed tomography: preliminary study. Oper. Neurosurg. 12(1), 4–13 (2015)
Rivaz, H., Collins, D.L.: Deformable registration of preoperative MR, pre-resection ultrasound, and post-resection ultrasound images of neurosurgery. Int. J. Comput. Assist. Radiol. Surg. 10(7), 1017–1028 (2015). https://doi.org/10.1007/s11548-014-1099-4
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Sanai, N., Polley, M.Y., McDermott, M.W., Parsa, A.T., Berger, M.S.: An extent of resection threshold for newly diagnosed glioblastomas: clinical article. J. Neurosurg. JNS 115(1), 3–8 (2011)
Shotton, J., Glocker, B., Zach, C., Izadi, S., Criminisi, A., Fitzgibbon, A.W.: Scene coordinate regression forests for camera relocalization in RGB-D images. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2930–2937 (2013)
Skandarani, Y., Jodoin, P.M., Lalande, A.: GANs for medical image synthesis: an empirical study. J. Imaging 9(3), 69 (2023)
Stoyanov, D.: Surgical vision. Ann. Biomed. Eng. 40, 332–345 (2012). https://doi.org/10.1007/s10439-011-0441-z
Sun, K., Pheiffer, T., Simpson, A., Weis, J., Thompson, R., Miga, M.: Near real-time computer assisted surgery for brain shift correction using biomechanical models. IEEE Transl. Eng. Health Med. 2, 1–13 (2014)
Tian, L., Lee, Y.Z., San José Estépar, R., Niethammer, M.: LiftReg: limited angle 2D/3D deformable registration. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) MICCAI 2022. LNCS, vol. 13436, pp. 207–216. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16446-0_20
Acknowledgement
The authors were partially supported by the following National Institutes of Health grants: R01EB027134, R03EB032050, R01EB032387, and R01EB034223.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Haouchine, N. et al. (2023). Learning Expected Appearances for Intraoperative Registration During Neurosurgery. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14228. Springer, Cham. https://doi.org/10.1007/978-3-031-43996-4_22
Download citation
DOI: https://doi.org/10.1007/978-3-031-43996-4_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43995-7
Online ISBN: 978-3-031-43996-4
eBook Packages: Computer ScienceComputer Science (R0)
