Advertisement

Probe-Based Rapid Hybrid Hyperspectral and Tissue Surface Imaging Aided by Fully Convolutional Networks

  • Jianyu LinEmail author
  • Neil T. Clancy
  • Xueqing Sun
  • Ji Qi
  • Mirek Janatka
  • Danail Stoyanov
  • Daniel S. Elson
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9902)

Abstract

Tissue surface shape and reflectance spectra provide rich intra-operative information useful in surgical guidance. We propose a hybrid system which displays an endoscopic image with a fast joint inspection of tissue surface shape using structured light (SL) and hyperspectral imaging (HSI). For SL a miniature fibre probe is used to project a coloured spot pattern onto the tissue surface. In HSI mode standard endoscopic illumination is used, with the fibre probe collecting reflected light and encoding the spatial information into a linear format that can be imaged onto the slit of a spectrograph. Correspondence between the arrangement of fibres at the distal and proximal ends of the bundle was found using spectral encoding. Then during pattern decoding, a fully convolutional network (FCN) was used for spot detection, followed by a matching propagation algorithm for spot identification. This method enabled fast reconstruction (12 frames per second) using a GPU. The hyperspectral image was combined with the white light image and the reconstructed surface, showing the spectral information of different areas. Validation of this system using phantom and ex vivo experiments has been demonstrated.

Keywords

Structured light Hyperspectral imaging Endoscopy Deep learning 

Notes

Acknowledgements

This work is funded by ERC 242991 and an Imperial College Confidence in Concept award. Jianyu Lin is supported by IGHI scholarship. Neil Clancy is supported by Imperial College Junior Research Fellowship. Danail Stoyanov is funded by EPSRC (EP/N013220/1, EP/N022750/1, EP/N027078/1, NS/A000027/1) and the EU-Horizon2020 (H2020-ICT-2015-688592).

References

  1. 1.
    Schwartz, J.J., Lichtenstein, G.R.: Magnification endoscopy, chromoendoscopy and other novel techniques in evaluation of patients with IBD. Tech. Gastrointest. Endosc. 6, 182–188 (2004)CrossRefGoogle Scholar
  2. 2.
    Maier-Hein, L., Mountney, P., Bartoli, A., Elhawary, H., Elson, D., Groch, A., Kolb, A., Rodrigues, M., Sorger, J., Speidel, S., Stoyanov, D.: Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery. Med. Image Anal. 17, 974–996 (2013)CrossRefGoogle Scholar
  3. 3.
    Lu, G., Fei, B.: Medical hyperspectral imaging: a Rev. J. Biomed. Opt. 19, 010901 (2014)CrossRefGoogle Scholar
  4. 4.
    Clancy, N.T., Arya, S., Stoyanov, D., Singh, M., Hanna, G.B., Elson, D.S.: Intraoperative measurement of bowel oxygen saturation using a multispectral imaging laparoscope. Biomed. Opt. Express 6, 4179–4190 (2015)CrossRefGoogle Scholar
  5. 5.
    Clancy, N.T., Arya, S., Corbett, R., Singh, M., Stoyanov, D., Crane, J.S., Duncan, N., Ebner, M., Caro, C.G., Hanna, G., Elson, D.S.: Optical measurement of anastomotic oxygenation dynamics. In: Biomedical Optics 2014, BS3A.23. Optical Society of America (2014)Google Scholar
  6. 6.
    Akbari, H., Halig, L.V., Schuster, D.M., Osunkoya, A., Master, V., Nieh, P.T., Chen, G.Z., Fei, B.: Hyperspectral imaging and quantitative analysis for prostate cancer detection. J. Biomed. Opt. 17, 0760051–07600510 (2012)CrossRefGoogle Scholar
  7. 7.
    Clancy, N.T., Lin, J., Arya, S., Hanna, G.B., Elson, D.S.: Dual multispectral and 3D structured light laparoscope. In: Proceedings of SPIE, Multimodal Biomedical Imaging X, vol. 9316, p. 93160C (2015)Google Scholar
  8. 8.
    Clancy, N.T., Stoyanov, D., James, D.R.C., Di Marco, A., Sauvage, V., Clark, J., Yang, G.-Z., Elson, D.S.: Multispectral image alignment using a three channel endoscope in vivo during minimally invasive surgery. Biomed. Opt. Express 3, 2567–2578 (2012)CrossRefGoogle Scholar
  9. 9.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  10. 10.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015, Part III. LNCS, vol. 9351, pp. 234–241. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-24574-4_28CrossRefGoogle Scholar
  11. 11.
    Clancy, N.T., Stoyanov, D., Maier-Hein, L., Groch, A., Yang, G.-Z., Elson, D.S.: Spectrally encoded fiber-based structured lighting probe for intraoperative 3D imaging. Biomedical Optics Express 2, 3119–3128 (2011)CrossRefGoogle Scholar
  12. 12.
    Maier-Hein, L., Groch, A., Bartoli, A., Bodenstedt, S., Boissonnat, G., Chang, P.L., Clancy, N.T., Elson, D.S., Haase, S., Heim, E., Hornegger, J., Jannin, P., Kenngott, H., Kilgus, T., Muller-Stich, B., Oladokun, D., Rohl, S., dos Santos, T.R., Schlemmer, H.P., Seitel, A., Speidel, S., Wagner, M., Stoyanov, D.: Comparative validation of single-shot optical techniques for laparoscopic 3-D surface reconstruction. IEEE Trans. Med. Imaging 33, 1913–1930 (2014)CrossRefGoogle Scholar
  13. 13.
    Lin, J., Clancy, N.T., Elson, D.S.: An endoscopic structured light system using multispectral detection. Int. J. CARS 10, 1941–1950 (2015)CrossRefGoogle Scholar
  14. 14.
    Lin, J., T.Clancy, N., Stoyanov, D., Elson, D.S.: Tissue surface reconstruction aided by local normal information using a self-calibrated endoscopic structured light system. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015, Part I. LNCS, vol. 9349, pp. 405–412. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-24553-9_50CrossRefGoogle Scholar
  15. 15.
    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp. 675–678. ACM, Orlando (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 2.5 International License (http://creativecommons.org/licenses/by-nc/2.5/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Authors and Affiliations

  • Jianyu Lin
    • 1
    • 2
    Email author
  • Neil T. Clancy
    • 1
    • 3
  • Xueqing Sun
    • 1
    • 3
  • Ji Qi
    • 1
    • 3
  • Mirek Janatka
    • 4
    • 5
  • Danail Stoyanov
    • 4
    • 5
  • Daniel S. Elson
    • 1
    • 3
  1. 1.Hamlyn Centre for Robotic SurgeryImperial College LondonLondonUK
  2. 2.Department of ComputingImperial College LondonLondonUK
  3. 3.Department of Surgery and CancerImperial College LondonLondonUK
  4. 4.Centre for Medical Image ComputingUniversity College LondonLondonUK
  5. 5.Department of Computer ScienceUniversity College LondonLondonUK

Personalised recommendations