Abstract
Minimally invasive surgeries are complex to perform, and surgical outcomes are varied due to a limited view of the surgical scene. There is a lack of reliable vision systems that can identify and segment different tissue types intra-operatively. Here we introduce a novel approach towards overcoming this limitation. Our approach extracts geometric and spectral information captured by a miniaturized camera using deep learning algorithms. We have successfully implemented a deep neural network and trained it for in-situ labelling of soft and hard tissues acquired from clinically relevant cadaveric studies. Although presented and validated for knee arthroscopy our approach can be implemented across different endoscopic platforms. The intraoperative nature of tissue segmentation could be easily implemented in medical imaging systems for achieving better outcomes in minimally invasive procedures. In addition to segmenting different tissue types like, ACL, femur, tibia, cartilage, and meniscus, our network can also segment surgical tools present inside the knee cavity. The achieved dice similarity score for the tissue types femur, tibia, ACL, and meniscus were 0.91, 0.71, 0.39, and 0.62.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ali S et al (2021) Arthroscopic Multi-Spectral Scene Segmentation Using Deep Learning. arXiv preprint arXiv:2103.02465
Jonmohamadi Y et al (2020) Automatic segmentation of multiple structures in knee arthroscopy using deep learning. IEEE Access 8:51853–51861. https://doi.org/10.1109/ACCESS.2020.2980025
Jaiprakash A et al (2017) Orthopaedic surgeon attitudes towards current limitations and the potential for robotic and technological innovation in arthroscopic surgery. J Orthop Surg 25(1):230949901668499
Price AJ et al (2015) Evidence-based surgical training in orthopaedics: how many arthroscopies of the knee are needed to achieve consultant level performance? Bone Joint J 97(10):1309–1315
Prete FP et al (2018) Robotic versus laparoscopic minimally invasive surgery for rectal cancer: a systematic review and meta-analysis of randomized controlled trials. Ann Surg 267(6):1034–1046
Xia L et al (2019) National trends and disparities of minimally invasive surgery for localized renal cancer, 2010 to 2015. Urol Oncol: Semin Orig Investig 37(3):182.e17–182.e27
Malik J et al (2001) Contour and texture analysis for image segmentation. Int J Comput Vis 43(1):7–27
Suga A et al (2008) Object recognition and segmentation using SIFT and graph cuts. In: 2008 19th international conference on pattern recognition. IEEE
Maqbool S et al (2020) m2caiSeg: Semantic Segmentation of Laparoscopic Images using Convolutional Neural Networks. arXiv preprint arXiv:2008.10134
Scheikl PM et al (2020) Deep learning for semantic segmentation of organs and tissues in laparoscopic surgery. Curr Dir Biomed Eng 6(1):1–11
Kletz S et al (2019) Identifying surgical instruments in laparoscopy using deep learning instance segmentation. In: 2019 international conference on content-based multimedia indexing (CBMI). IEEE
Badrinarayanan V, Kendall A, Cipolla R (2017) Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pattern Anal Mach Intell 39(12):2481–2495
Diakogiannis FI et al (2020) ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data. ISPRS J Photogramm Remote Sens 162:94–114
Jha D et al (2019) Resunet++: an advanced architecture for medical image segmentation. In: 2019 IEEE international symposium on multimedia (ISM). IEEE
Sun J et al (2020) Saunet: shape attentive u-net for interpretable medical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, Cham
Ali S et al (2021) Supervised scene illumination control in stereo arthroscopes for robot assisted minimally invasive surgery. IEEE Sensors J 21(10):11577–11587
Liu F et al (2020) Self-supervised depth estimation to regularise semantic segmentation in knee arthroscopy. In: International conference on medical image computing and computer-assisted intervention. Springer, Cham
Stigell P, Miyata K, Hauta-Kasari M (2007) Wiener estimation method in estimating of spectral reflectance from RGB images. Pattern Recognit Image Anal 17(2):233–242
Chen S, Liu Q (2012) Modified Wiener estimation of diffuse reflectance spectra from RGB values by the synthesis of new colors for tissue measurements. J Biomed Opt 17(3):030501
Peng X et al (2017) Self-training-based spectral image reconstruction for art paintings with multispectral imaging. Appl Opt 56(30):8461
Otsu H, Yamamoto M, Hachisuka T (2018) Reproducing spectral reflectances from tristimulus colours: reproducing spectral reflectances from tristimulus colours. Comput Graph Forum 37(6):370–381
Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, Cham
Zhou Z et al (2018) Unet++: a nested u-net architecture for medical image segmentation. In: Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, Cham, pp 3–11
He K et al (2016) Identity mappings in deep residual networks. In: European conference on computer vision. Springer, Cham
Wang X et al (2018) Non-local neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition
Zhang J et al (2018) MDU-net: multi-scale densely connected u-net for biomedical image segmentation. arXiv preprint arXiv:1812.00352
Cheng J et al (2020) Fully convolutional attention network for biomedical image segmentation. Artif Intell Med 107:101899
Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition
Ali S et al (2021) Surgery Scene Restoration for Robot Assisted Minimally Invasive Surgery. arXiv preprint arXiv:2109.02253
Zhang Z, Liu Q, Wang Y (2018) Road extraction by deep residual u-net. IEEE Geosci Remote Sens Lett 15(5):749–753
Acknowledgements
This work is supported by the Australia-India Strategic Research Fund (Grant AISRF53820), the Medical Engineering Research Facility at QUT, and QUT’s Centre for Robotics. We acknowledge Dr. Yu Takeda, Dr. Fumio Sasazawa for the ground truth segmentation data used in this study.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Ali, S., Pandey, A.K. (2022). Towards Robotic Knee Arthroscopy: Spatial and Spectral Learning Model for Surgical Scene Segmentation. In: Uddin, M.S., Jamwal, P.K., Bansal, J.C. (eds) Proceedings of International Joint Conference on Advances in Computational Intelligence. Algorithms for Intelligent Systems. Springer, Singapore. https://doi.org/10.1007/978-981-19-0332-8_19
Download citation
DOI: https://doi.org/10.1007/978-981-19-0332-8_19
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-19-0331-1
Online ISBN: 978-981-19-0332-8
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)