Skip to main content

EasyLabels: weak labels for scene segmentation in laparoscopic videos

International Journal of Computer Assisted Radiology and Surgery volume 14, pages 1247–1257 (2019)Cite this article

Abstract

Purpose

We present a different approach for annotating laparoscopic images for segmentation in a weak fashion and experimentally prove that its accuracy when trained with partial cross-entropy is close to that obtained with fully supervised approaches.

Methods

We propose an approach that relies on weak annotations provided as stripes over the different objects in the image and partial cross-entropy as the loss function of a fully convolutional neural network to obtain a dense pixel-level prediction map.

Results

We validate our method on three different datasets, providing qualitative results for all of them and quantitative results for two of them. The experiments show that our approach is able to obtain at least \(90\%\) of the accuracy obtained with fully supervised methods for all the tested datasets, while requiring \(\sim 13\)\(\times \) less time to create the annotations compared to full supervision.

Conclusions

With this work, we demonstrate that laparoscopic data can be segmented using very few annotated data while maintaining levels of accuracy comparable to those obtained with full supervision.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Notes

  1. For simplicity, even if there are approaches either using points, bounding boxes or scribbles, we will refer to this approach as just “scribbles.”

  2. Please note that the skeleton is computed as the ridge of the distance transform.

References

  1. Bearman A, Russakovsky O, Ferrari V, Fei-Fei L (2016) What’s the point: semantic segmentation with point supervision. In: European conference on computer vision. Springer, pp 549–565

  2. Bodenstedt S, Allan M, Agustinos A, Du X, Garcia-Peraza-Herrera L, Kenngott H, Kurmann T, Müller-Stich B, Ourselin S, Pakhomov D, Sznitman R, Teichmann M, Thoma M, Vercauteren T, Voros S, Wagner M, Wochner P, Maier-Hein L, Stoyanov D, Speidel S. (2018) Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery. arXiv preprint arXiv:1805.02475

  3. Bodenstedt S, Ohnemus A, Katic D, Wekerle AL, Wagner M, Kenngott H, Müller-Stich B, Dillmann R, Speidel S. (2018) Real-time image-based instrument classification for laparoscopic surgery. arXiv preprint arXiv:1808.00178

  4. Bouget D, Allan M, Stoyanov D, Jannin P (2017) Vision-based and marker-less surgical tool detection and tracking: a review of the literature. Med Image Anal 35:633–654

    Article  PubMed  Google Scholar 

  5. Chen LC, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision (ECCV), pp 801–818

  6. Chollet F (2017) Xception: Deep learning with depthwise separable convolutions. In: The IEEE conference on computer vision and pattern recognition (CVPR)

  7. Gao M, Xu Z, Lu L, Wu A, Nogues I, Summers RM, Mollura DJ (2016) Segmentation label propagation using deep convolutional neural networks and dense conditional random field. In: 2016 IEEE 13th international symposium on biomedical imaging (ISBI). IEEE, pp 1265–1268

  8. García-Peraza-Herrera LC, Li W, Fidon L, Gruijthuijsen C, Devreker A, Attilakos G, Deprest J, Vander Poorten E, Stoyanov D, Vercauteren T, Ourselin S (2017) Toolnet: holistically-nested real-time segmentation of robotic surgical tools. In: 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 5717–5722

  9. García-Peraza-Herrera LC, Li W, Gruijthuijsen C, Devreker A, Attilakos G, Deprest J, Vander Poorten E, Stoyanov D, Vercauteren T, Ourselin S (2016) Real-time segmentation of non-rigid surgical tools based on deep learning and tracking. In: International workshop on computer-assisted and robotic endoscopy. Springer, pp 84–95

  10. Laina I, Rieke N, Rupprecht C, Vizcaíno JP, Eslami A, Tombari F, Navab N (2017) Concurrent segmentation and localization for tracking of surgical instruments. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 664–672

  11. Lejeune L, Grossrieder J, Sznitman R (2018) Iterative multi-path tracking for video and volume segmentation with sparse point supervision. Med Image Anal 50:65–81

    Article  PubMed  Google Scholar 

  12. Lin D, Dai J, Jia J, He K, Sun J (2016) Scribblesup: scribble-supervised convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3159–3167

  13. Maier-Hein L, Ross T, Gröhl J, Glocker B, Bodenstedt S, Stock C, Heim E, Götz M, Wirkert S, Kenngott H, Speidel S, Maier-Hein K (2016) Crowd-algorithm collaboration for large-scale endoscopic image annotation with confidence. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 616–623

  14. MICCAI 2015: Endovis 2015 instrument segmentation and tracking. https://endovissub-instrument.grand-challenge.org (2015). [Online; Accessed 6-Nov-2018]

  15. Pakhomov D, Premachandran V, Allan M, Azizian M, Navab N (2017) Deep residual learning for instrument segmentation in robotic surgery. arXiv preprint arXiv:1703.08580

  16. Ross T, Zimmerer D, Vemuri A, Isensee F, Wiesenfarth M, Bodenstedt S, Both F, Kessler P, Wagner M, Müller B, Kengott H, Speidel S, Kop-Schneider A, Maier-Hein K, Maier-Hein L (2018) Exploiting the potential of unlabeled endoscopic video data with self-supervised learning. Int J Comput Assist Radiol Surg 13:1–9

    Article  Google Scholar 

  17. Schoeffmann K, Husslein H, Kletz S, Petscharnig S, Muenzer B, Beecks C (2017) Video retrieval in laparoscopic video recordings with dynamic content descriptors. Multimed Tools Appl 77:16813–16832. https://doi.org/10.1007/s11042-017-5252-2

    Article  Google Scholar 

  18. Shvets A, Rakhlin A, Kalinin AA, Iglovikov V (2018) Automatic instrument segmentation in robot-assisted surgery using deep learning. arXiv preprint arXiv:1803.01207

  19. Stoyanov D (2012) Surgical vision. Ann Biomed Eng 40(2):332–345

    Article  PubMed  Google Scholar 

  20. Tang M, Djelouah A, Perazzi F, Boykov Y, Schroers C (2018) Normalized cut loss for weakly-supervised cnn segmentation. In: IEEE conference on computer vision and pattern recognition (CVPR), Salt Lake City

  21. Tang P, Wang X, Wang A, Yan Y, Liu W, Huang J, Yuille A (2018) Weakly supervised region proposal network and object detection. In: Proceedings of the European conference on computer vision (ECCV), pp 352–368

  22. Vardazaryan A, Mutter D, Marescaux J, Padoy N (2018) Weakly-supervised learning for tool localization in laparoscopic videos. In: Stoyanov D et al (eds) Intravascular imaging and computer assisted stenting and large-scale annotation of biomedical data and expert label synthesis. LABELS 2018, CVII 2018, STENT 2018. Lecture Notes in Computer Science, vol 11043. Springer, Cham, pp 169–179

  23. Wang X, You S, Li X, Ma H (2018) Weakly-supervised semantic segmentation by iteratively mining common object features. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1354–1362

  24. Zhao X, Liang S, Wei Y (2018) Pseudo mask augmented object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4061–4070

Download references

Author information

Affiliations

  1. Digital Surgery, London, United Kingdom

    Félix Fuentes-Hurtado, Abdolrahim Kadkhodamohammadi, Evangello Flouty, Santiago Barbarisi, Imanol Luengo & Danail Stoyanov

  2. Wellcome/ESPRC Centre for Interventional and Surgical Sciences, London, United Kingdom

    Danail Stoyanov

Authors
  1. Félix Fuentes-Hurtado
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Abdolrahim Kadkhodamohammadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Evangello Flouty
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Santiago Barbarisi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Imanol Luengo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Danail Stoyanov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Félix Fuentes-Hurtado.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

For this type of study, formal consent is not required.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Fuentes-Hurtado, F., Kadkhodamohammadi, A., Flouty, E. et al. EasyLabels: weak labels for scene segmentation in laparoscopic videos. Int J CARS 14, 1247–1257 (2019). https://doi.org/10.1007/s11548-019-02003-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-019-02003-2

Keywords

  • Computer-assisted interventions
  • Laparoscopy
  • Instrument detection and segmentation