Automated Polyp Segmentation in Colonoscopy Frames Using Fully Convolutional Neural Network and Textons

  • Lei ZhangEmail author
  • Sunil Dolwani
  • Xujiong Ye
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 723)


In this paper, we presented a novel hybrid classification based method for fully automated polyp segmentation in colonoscopy video frames. It contains two main steps: initial region proposals generation and regions refinement. Both machine learned features and hand crafted features are taken into account for polyp segmentation. More specifically, the hierarchical features of polyps are learned by fully convolutional neural network (FCN), while the context information related to the polyp boundaries is modeled by texton patch representation. The FCN provides pixel-wise prediction and initial polyp region candidates. Those candidates are further refined by patch-wise classification using texton based spatial features and a random forest classifier. The segmentation results are evaluated on a publicly available CVC-ColonDB database. On average, our method achieves 97.54% of accuracy, 75.66% of sensitivity, 98.81% of specificity and DICE of 0.70%. The fast execution time (0.16 s/frame) demonstrates the promise of our method to be used in real-time clinical colonoscopic examination.


Optical colonoscopy Polyp segmentation Fully convolutional neural network (FCN) Textons Random forest classifier 



This research was supported by Cancer Research UK (CRUK) funded project “Bowels - inside out” (A22873).


  1. 1.
    Winawer, S.J., Zauber, A.G., Ho, M.N., Obrien, M.J., Gottlieb, L.S., Sternberg, S.S., Waye, J.D., Schapiro, M., Bond, J.H., Panish, J.F., Ackroyd, F., Shike, M., Kurtz, R.C., Hornsbylewis, L., Gerdes, H., Stewart, E.T.: Prevention of colorectal-cancer by colonoscopic polypectomy. New Engl. J. Med. 329, 1977–1981 (1993)CrossRefGoogle Scholar
  2. 2.
    Lieberman, D.: Quality and colonoscopy: a new imperative. Gastrointest. Endosc. 61, 392–394 (2005)CrossRefGoogle Scholar
  3. 3.
    Tajbakhsh, N., Gurudu, S.R., Liang, J.: A classification-enhanced vote accumulation scheme for detecting colonic polyps. In: Yoshida, H., Warfield, S., Vannier, M.W. (eds.) ABD-MICCAI 2013. LNCS, vol. 8198, pp. 53–62. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-41083-3_7 CrossRefGoogle Scholar
  4. 4.
    Hwang, S., Oh, J., Tavanapong, W., Wong, J., de Groen, P.C.: Polyp detection in colonoscopy video using elliptical shape feature. In: IEEE International Conference on Image Processing, pp. 1029–1032 (2007)Google Scholar
  5. 5.
    Bernal, J., Sanchez, J., Vilarino, F.: Towards automatic polyp detection with a polyp appearance model. Pattern Recogn. 45, 3166–3182 (2012)CrossRefGoogle Scholar
  6. 6.
    Li, P., Chan, K.L., Krishnan, S.M.: Learning a multi-size patch-based hybrid kernel machine ensemble for abnormal region detection in colonoscopic images. In: Proceedings of CVPR, pp. 670–675. IEEE (2005)Google Scholar
  7. 7.
    Park, S.Y., Sargent, D., Spofford, I., Vosburgh, K.G., A-Rahim, Y.: A colon video analysis framework for polyp detection. IEEE Trans. Bio-Med. Eng. 59, 1408–1418 (2012)CrossRefGoogle Scholar
  8. 8.
    Bae, S.H., Yoon, K.J.: Polyp detection via imbalanced learning and discriminative feature learning. IEEE Trans. Bio-Med. Imaging 34, 2379–2393 (2015)CrossRefGoogle Scholar
  9. 9.
    Tajbakhsh, N., Gurudu, S.R., Liang, J.: Automatic polyp detection using global geometric constraints and local intensity variation patterns. In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8674, pp. 179–187. Springer, Cham (2014). doi: 10.1007/978-3-319-10470-6_23 Google Scholar
  10. 10.
    Tajbakhsh, N., Gurudu, S.R., Liang, J.M.: Automatic polyp detection in colonoscopy videos using an ensemble of convolutional neural networks. In: International symposium on Biomed Imaging, pp. 79–83 (2015)Google Scholar
  11. 11.
    Tajbakhsh, N., Gurudu, S.R., Liang, J.M.: Automated polyp detection in colonoscopy videos using shape and context information. IEEE Trans. Med. Imaging 35, 630–644 (2016)CrossRefGoogle Scholar
  12. 12.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431–3440 (2015)Google Scholar
  13. 13.
    Breiman, L.: Random forests. Mach. Learn. 45, 5–32 (2001)CrossRefzbMATHGoogle Scholar
  14. 14.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)CrossRefGoogle Scholar
  15. 15.
    LeCun, Y., Kavukcuoglu, K., Farabet, C.: Convolutional networks and applications in vision. In: Ieee International Symposium on Circuits and systems, pp. 253–256 (2010)Google Scholar
  16. 16.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: CORR abs/1409.1556 (2014)Google Scholar
  17. 17.
    Tajbakhsh, N., Shin, J.Y., Gurudu, S.R., Hurst, R.T., Kendall, C.B., Gotway, M.B., Jianming, L.: Convolutional neural networks for medical image analysis: full training or fine tuning? IEEE Trans. Med. Imaging 35, 1299–1312 (2016)CrossRefGoogle Scholar
  18. 18.
    Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). doi: 10.1007/978-3-319-10590-1_53 Google Scholar
  19. 19.
    Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 580–587 (2014)Google Scholar
  20. 20.
    Varma, M., Zisserman, A.: A statistical approach to texture classification from single images. Int. J. Comput. Vis. 62, 61–81 (2005)CrossRefGoogle Scholar
  21. 21.
    Leung, T., Malik, J.: Representing and recognizing the visual appearance of materials using three-dimensional textons. Int. J. Comput. Vis. 43, 29–44 (2001)CrossRefzbMATHGoogle Scholar
  22. 22.
    Daugman, J.G.: Uncertainty relation for resolution in space, spatial-frequency, and orientation optimized by two-dimensional visual cortical filters. J. Opt. Soc. Am. 2, 1160–1169 (1985)CrossRefGoogle Scholar
  23. 23.
    Dice, L.R.: Measures of the amount of ecologic association between species. Ecology 26, 297–302 (1945)CrossRefGoogle Scholar
  24. 24.
    Vedaldi, A., Lenc, K.: MatConvNet convolutional neural networks for MATLAB. In: MM 2015: Proceedings of the 2015 Acm Multimedia Conference, pp. 689–692 (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Laboratory of Vision Engineering, Computer ScienceUniversity of LincolnLincolnUK
  2. 2.Division of Population Medicine, School of MedicineCardiff UniversityCardiffUK

Personalised recommendations