A novel summary report of colonoscopy: timeline visualization providing meaningful colonoscopy video information

  • Minwoo Cho
  • Jee Hyun Kim
  • Hyoun Joong Kong
  • Kyoung Sup HongEmail author
  • Sungwan KimEmail author
Original Article



The colonoscopy adenoma detection rate depends largely on physician experience and skill, and overlooked colorectal adenomas could develop into cancer. This study assessed a system that detects polyps and summarizes meaningful information from colonoscopy videos.


One hundred thirteen consecutive patients had colonoscopy videos prospectively recorded at the Seoul National University Hospital. Informative video frames were extracted using a MATLAB support vector machine (SVM) model and classified as bleeding, polypectomy, tool, residue, thin wrinkle, folded wrinkle, or common. Thin wrinkle, folded wrinkle, and common frames were reanalyzed using SVM for polyp detection. The SVM model was applied hierarchically for effective classification and optimization of the SVM.


The mean classification accuracy according to type was over 93%; sensitivity was over 87%. The mean sensitivity for polyp detection was 82.1%, and the positive predicted value (PPV) was 39.3%. Polyps detected using the system were larger (6.3 ± 6.4 vs. 4.9 ± 2.5 mm; P = 0.003) with a more pedunculated morphology (Yamada type III, 10.2 vs. 0%; P < 0.001; Yamada type IV, 2.8 vs. 0%; P < 0.001) than polyps missed by the system. There were no statistically significant differences in polyp distribution or histology between the groups. Informative frames and suspected polyps were presented on a timeline. This summary was evaluated using the system usability scale questionnaire; 89.3% of participants expressed positive opinions.


We developed and verified a system to extract meaningful information from colonoscopy videos. Although further improvement and validation of the system is needed, the proposed system is useful for physicians and patients.


Colonoscopy Polyp detection Summary report Visualization Classification 



This work was supported by Grant No. 0320150450 from the Seoul National University Hospital.

Compliance with ethical standards

This study was approved by the institutional review board of the Seoul National University Hospital (IRB No. 1509-062-703, approval date July 13, 2016), and it was conducted in accordance with the Declaration of Helsinki. Written informed consent was obtained from all participants.

Conflict of interest

The authors declare that they have no conflict of interest.

Supplementary material

384_2018_2980_MOESM1_ESM.docx (13 kb)
ESM 1 (DOCX 12 kb)


  1. 1.
    Haggar FA, Boushey RP (2009) Colorectal cancer epidemiology: incidence, mortality, survival, and risk factors. Clin Colon Rectal Surg 22:191–197CrossRefPubMedPubMedCentralGoogle Scholar
  2. 2.
    Quintero E, Hassan C, Senore C et al (2012) Progress and challenges in colorectal cancer screening. Gastroenterol Res Pract 2012:846985PubMedPubMedCentralGoogle Scholar
  3. 3.
    Rex DK (2002) Rationale for colonoscopy screening and estimated effectiveness in clinical practice. Gastrointest Endosc Clin N Am 12(1):65–75CrossRefPubMedGoogle Scholar
  4. 4.
    Leslie A, Carey F, Pratt N et al (2002) The colorectal adenoma–carcinoma sequence. Br J Surg 89:845–860CrossRefPubMedGoogle Scholar
  5. 5.
    Simmons DT et al (2006) Impact of endoscopist withdrawal speed on polyp yield: implications for optimal colonoscopy withdrawal time. Aliment Pharmacol Ther 24(6):965–971CrossRefPubMedGoogle Scholar
  6. 6.
    Anderson BW, Smyrk TC, Anderson KS, Mahoney DW, Devens ME, Sweetser SR, Kisiel JB, Ahlquist DA (2016) Endoscopic overestimation of colorectal polyp size. Gastrointest Endosc 83(1):201–208CrossRefPubMedGoogle Scholar
  7. 7.
    Burggraaf J, Kamerling IMC, Gordon PB, Schrier L, de Kam ML, Kales AJ, Bendiksen R, Indrevoll B, Bjerke RM, Moestue SA, Yazdanfar S, Langers AMJ, Swaerd-Nordmo M, Torheim G, Warren MV, Morreau H, Voorneveld PW, Buckle T, van Leeuwen FWB, Ødegårdstuen L-I, Dalsgaard GT, Healey A, Hardwick JCH (2015) Detection of colorectal polyps in humans using an intravenously administered fluorescent peptide targeted against c-Met. Nat Med 21(8):955–961CrossRefPubMedGoogle Scholar
  8. 8.
    Nishihara R et al (2013) Long-term colorectal-cancer incidence and mortality after lower endoscopy. N Engl J Med 369(12):1095–1105CrossRefPubMedGoogle Scholar
  9. 9.
    Hong SN, Sung IK, Kim JH, Choe WH, Kim BK, Ko SY, Lee JH, Seol DC, Ahn SY, Lee SY, Park HS, Shim CS (2012) The effect of the bowel preparation status on the risk of missing polyp and adenoma during screening colonoscopy: a tandem colonoscopic study. Clinical endoscopy 45:404–411CrossRefPubMedPubMedCentralGoogle Scholar
  10. 10.
    Kaz AM, Dominitz JA (2017) The name game: circumventing quality metrics by categorizing incomplete colonoscopy as sigmoidoscopy. Am J Gastroenterol 112:ajg2017258CrossRefGoogle Scholar
  11. 11.
    De Jonge V et al (2012) Quality evaluation of colonoscopy reporting and colonoscopy performance in daily clinical practice. Gastrointest Endosc 75(1):98–106CrossRefPubMedGoogle Scholar
  12. 12.
    Wolf HJ et al (2015) Colon cancer screening for Colorado’s underserved: a community clinic/academic partnership. Am J Prev Med 48(3):264–270CrossRefPubMedGoogle Scholar
  13. 13.
    Shah-Ghassemzadeh NK et al (2017) Training mid-career internists to perform high-quality colonoscopy: a pilot training programme to meet increasing demands for colonoscopy. Postgrad Med J 93(1102):484–488CrossRefPubMedGoogle Scholar
  14. 14.
    Adler A et al (2013) Factors determining the quality of screening colonoscopy: a prospective study on adenoma detection rates, from 12 134 examinations (Berlin colonoscopy project 3, BECOP-3). Gut 62(2):236–241CrossRefPubMedGoogle Scholar
  15. 15.
    Gurudu SR, Ratuapli SK, Leighton JA, Heigh RI, Crowell MD (2011) Adenoma detection rate is not influenced by the timing of colonoscopy when performed in half-day blocks. Am J Gastroenterol 106(8):1466–1471CrossRefPubMedGoogle Scholar
  16. 16.
    Serper M et al (2014) Patient factors that affect quality of colonoscopy preparation. Clin Gastroenterol Hepatol 12(3):451–457CrossRefPubMedGoogle Scholar
  17. 17.
    Sanaka MR, Deepinder F, Thota PN, Lopez R, Burke CA (2009) Adenomas are detected more often in morning than in afternoon colonoscopy. Am J Gastroenterol 104(7):1659–1664CrossRefPubMedGoogle Scholar
  18. 18.
    Lee TJW et al (2014) Colonoscopic factors associated with adenoma detection in a national colorectal cancer screening program. Endoscopy 46(03):203–211CrossRefPubMedGoogle Scholar
  19. 19.
    Shang-bo, Zhou, et al. (2016) Bleeding detection in wireless capsule endoscope based on color feature vector. DEStech Transactions on Computer Science and Engineering, aice-ncsGoogle Scholar
  20. 20.
    Brooker JC, Saunders BP, Shah SG, Thapar CJ, Suzuki N, Williams CB (2002) Treatment with argon plasma coagulation reduces recurrence after piecemeal resection of large sessile colonic polyps: a randomized trial and recommendations. Gastrointest Endosc 55(3):371–375CrossRefPubMedGoogle Scholar
  21. 21.
    Gupta S et al (2011) Development and validation of a novel method for assessing competency in polypectomy: direct observation of polypectomy skills. Gastrointest Endosc 73(6):1232–1239. e2CrossRefPubMedGoogle Scholar
  22. 22.
    Kaneoka AS, Langmore SE, Krisciunas GP, Field K, Scheel R, McNally E, Walsh MJ, O'Dea MB, Cabral H (2013) The Boston residue and clearance scale: preliminary reliability and validity testing. Folia Phoniatr Logop 65(6):312–317CrossRefPubMedGoogle Scholar
  23. 23.
    Münzer B, Schoeffmann K, Böszörmenyi L (2017) Content-based processing and analysis of endoscopic images and videos: a survey. Multimed Tools and Appl, 1–40Google Scholar
  24. 24.
    Sudha N, Santhiyakumari N (2016) MIEEE. Summarizing wireless capsule endoscopy video frame based on inter-frame structural similarity index. Adv Nat Appl Sci 10(6 SE):199–208Google Scholar
  25. 25.
    Terada T (2015) Histopathological study using computer database of 10 000 consecutive gastric specimens:(1) benign conditions. Gastroenterol Rep 3(3):238–242CrossRefGoogle Scholar
  26. 26.
    Hu E, Sakanashi H, Nosato H, Takahashi E, Suzuki Y, Takeuchi K, Aoki H, Murakawa M (2016) Bleeding and tumor detection for capsule endoscopy images using improved geometric feature. J Med Biol Eng 36(3):344–356CrossRefGoogle Scholar
  27. 27.
    Yuan Y, Li B, Meng MQ-H (2016) Improved bag of feature for automatic polyp detection in wireless capsule endoscopy images. IEEE Trans Autom Sci Eng 13(2):529–535CrossRefGoogle Scholar
  28. 28.
    Salvi E et al. (2017) Exploring IBM Watson to extract meaningful information from the list of references of a clinical practice guideline. In: Conference on Artificial Intelligence in Medicine in Europe. Springer: Cham, p. 193–197Google Scholar
  29. 29.
    Diomaiuta C et al. (2017) A novel system for the automatic extraction of a patient problem summary. In: Computers and Communications (ISCC), 2017 I.E. Symposium on IEEE. p. 182–186Google Scholar
  30. 30.
    Hoyt RE et al (2016) IBM Watson analytics: automating visualization, descriptive, and predictive statistics. JMIR Public Health Surveill 2:2CrossRefGoogle Scholar
  31. 31.
    Cruz JA, Wishart DS (2006) Applications of machine learning in cancer prediction and prognosis. Cancer Informat 2:59CrossRefGoogle Scholar
  32. 32.
    Ahn M, Choi M, Kim Y (2016) Factors associated with the timeliness of electronic nursing documentation. Healthc Inform Res 22:270–276CrossRefPubMedPubMedCentralGoogle Scholar
  33. 33.
    Ikematsu H, Saito Y, Tanaka S, Uraoka T, Sano Y, Horimatsu T, Matsuda T, Oka S, Higashi R, Ishikawa H, Kaneko K (2012) The impact of narrow band imaging for colon polyp detection: a multicenter randomized controlled trial by tandem colonoscopy. J Gastroenterol 47:1099–1107CrossRefPubMedGoogle Scholar
  34. 34.
    Vilarino F et al. (2007) Automatic labeling of colonoscopy video for cancer detection. Pattern Recognit Image Anal 290–297Google Scholar
  35. 35.
    Fallah M, Niakan K, Sharareh R (2017) Systematic review of data mining applications in patient-centered mobile-based information systems. Healthc Inform Res 23(4):262–270CrossRefPubMedPubMedCentralGoogle Scholar
  36. 36.
    Hoyos J (2017) Automatic classification of non-informative frames in colonoscopy videos using texture analysis. In: Progress in pattern recognition, image analysis, computer vision, and applications: 21st Iberoamerican Congress, CIARP 2016, Lima, Peru, November 8–11, 2016, proceedings. Spring, p. 401Google Scholar
  37. 37.
    Park BE, Jang WS, Yoo SK (2016) Texture analysis of supraspinatus ultrasound image for computer aided diagnostic system. Healthc Inform Res 22:299–304CrossRefPubMedPubMedCentralGoogle Scholar
  38. 38.
    Timotius IK et al (2012) Abnormality detection for capsule endoscope images based on support vector machines. Biomed Eng Appl Basis Commun 24(01):71–83CrossRefGoogle Scholar
  39. 39.
    Xiong Y, et al. (2015) Bleeding detection in wireless capsule endoscopy based on mst clustering and svm. In: Signal processing systems (SiPS), 2015 I.E. Workshop on IEEE. p. 1–4Google Scholar
  40. 40.
    Mackiewicz MW, Fisher M, Jamieson C (2008) Bleeding detection in wireless capsule endoscopy using adaptive colour histogram model and support vector classification. In: Medical imaging. International Society for Optics and Photonics. p. 69140R-69140R-12Google Scholar
  41. 41.
    Hwang, Sae, et al. (2007) Polyp detection in colonoscopy video using elliptical shape feature. In: Image processing, 2007. ICIP 2007. IEEE International Conference on. IEEE. p. II-465-II-468Google Scholar
  42. 42.
    Nagtegaal ID (2015) Current concepts of colorectal cancer resection pathology. Histopathology 66(1):102–111CrossRefPubMedGoogle Scholar
  43. 43.
    Sonka M, Hlavac V, Boyle R (2014) Image processing, analysis, and machine vision. Cengage Learning, BostonGoogle Scholar
  44. 44.
    Hua BO, Fu-Long M, Li-Cheng J (2006) Research on computation of GLCM of image texture [J]. Acta Electron Sin 1(1):155–158Google Scholar
  45. 45.
    Iakovidis DK, Maroulis DE, Karkanis SA (2006) An intelligent system for automatic detection of gastrointestinal adenomas in video endoscopy. Comput Biol Med 36(10):1084–1103CrossRefPubMedGoogle Scholar
  46. 46.
    Bae S-H, Yoon K-J (2015) Polyp detection via imbalanced learning and discriminative feature learning. IEEE Trans Med Imaging 34(11):2379–2393CrossRefPubMedGoogle Scholar
  47. 47.
    Ionescu M et al (2015) Methods of diagnosis of intestinal polyps with software applications in wireless capsule endoscopy. Ann Comput Sci Ser 13:1Google Scholar
  48. 48.
    Valdeavilla EB, Miaou S-G Classification of capsule endoscope images using local binary patterns and support vector machinesGoogle Scholar
  49. 49.
    Rastogi A, Bansal A, Wani S, Callahan P, McGregor DH, Cherian R, Sharma P (2008) Narrow-band imaging colonoscopy--a pilot feasibility study for the detection of polyps and correlation of surface patterns with polyp histologic diagnosis. Gastrointest Endosc 67:280–286CrossRefPubMedGoogle Scholar
  50. 50.
    Kominami Y et al (2016) Computer-aided diagnosis of colorectal polyp histology by using a real-time image recognition system and narrow-band imaging magnifying colonoscopy. Gastrointest Endosc 83(3):643–649CrossRefPubMedGoogle Scholar
  51. 51.
    Oh ST (2003) Colorectal polyps: endoscopic diagnosis and polypectomy. Journal of the Korean Medical Association 46(7):594–604CrossRefGoogle Scholar
  52. 52.
    Sosna J et al (2003) CT colonography of colorectal polyps: a metaanalysis. Am J Roentgenol 181(6):1593–1598CrossRefGoogle Scholar
  53. 53.
    Brooke J (1996) SUS-A quick and dirty usability scale. Usability evaluation in industry. 189:4–7Google Scholar
  54. 54.
    Liu D-Y et al (2016) Identification of lesion images from gastrointestinal endoscope based on feature extraction of combinational methods with and without learning process. Med Image Anal 32:281–294CrossRefPubMedGoogle Scholar
  55. 55.
    Wu X, Chen H, Gan T, Chen J, Ngo C-W, Peng Q (2016) Automatic hookworm detection in wireless capsule endoscopy images. IEEE Trans Med Imaging 35(7):1741–1752CrossRefPubMedGoogle Scholar
  56. 56.
    Wang S et al (2016) Computer-aided endoscopic diagnosis without human-specific labeling. IEEE Trans Biomed Eng 63(11):2347–2358CrossRefPubMedGoogle Scholar
  57. 57.
    Davila RE, Rajan E, Baron TH (2006) ASGE guideline: colorectal cancer screening and surveillance. Gastrointest Endosc 63:546–557CrossRefPubMedGoogle Scholar
  58. 58.
    Sanaka MR, Super DM, Feldman ES, Mullen KD, Ferguson D, McCullough AJ (2006) Improving compliance with postpolypectomy surveillance guidelines: an interventional study using a continuous quality improvement initiative. Gastrointest Endosc 63:97–103CrossRefPubMedGoogle Scholar
  59. 59.
    Wang Y, Tavanapong W, Wong J, Oh J, De Groen PC (2014) Part-based multiderivative edge cross-sectional profiles for polyp detection in colonoscopy. IEEE J Biomed Health Inform 18:1379–1389CrossRefPubMedGoogle Scholar
  60. 60.
    Amber A et al. (2015) Feature point based polyp tracking in endoscopic videos. In: Applied computing and information technology/2nd International Conference on Computational Science and Intelligence (ACIT-CSI), 2015 3rd International Conference on IEEE. p. 299–304.Google Scholar
  61. 61.
    Park S, Lee M, Kwak N (2015) Polyp detection in colonoscopy videos using deeply-learned hierarchical features. Seoul National University, SeoulGoogle Scholar
  62. 62.
    Yu L, Chen H, Dou Q, Qin J, Heng PA (2016) Integrating online and offline 3D deep learning for automated polyp detection in colonoscopy videos. IEEE J Biomed Health InformGoogle Scholar
  63. 63.
    Bernal J, Sanchez FJ, Fernandez-Esparrach G, Gil D, Rodrıguez C, Vilarino F (2015) WM-DOVA maps for accurate polyp highlighting in colonoscopy: validation vs. saliency maps from physicians. Comput Med Imag Graph 43:99–111CrossRefGoogle Scholar
  64. 64.
    Tajbakhsh N, Gurudu SR, Liang J (2016) Automated polyp detection in colonoscopy videos using shape and context information. IEEE Trans Med Imag 35(2):630–644CrossRefGoogle Scholar
  65. 65.
    N. Tajbakhsh, S. R. Gurudu, J. Liang (2015) Automatic polyp detection in colonoscopy videos using an ensemble of convolutional neural networks. In: Proc. 2015 I.E. 12th Int Symp Biomed Imag, pp. 79–83Google Scholar
  66. 66.
    Sousa AC, Figueiredo P, Lopes S, Gouveia H, Sofia C, Correia LM (2007) A thousand total colonoscopies: what is the relationship between distal and proximal findings? Acta Medica Port 21:461–466Google Scholar
  67. 67.
    Silva SM, Rosa VF, Santos AC, Almeida RM, Oliveira PG, Sousa JB (2014) Influence of patient age and colorectal polyp size on histopathology findings. Arq Bras Cir Dig 27:109–113CrossRefPubMedPubMedCentralGoogle Scholar
  68. 68.
    Hochberger J, Maiss J, Magdeburg B, Cohen J, Hahn EG (2001) Training simulators and education in gastrointestinal endoscopy: current status and perspectives in 2001. Endoscopy 33(06):541–549CrossRefPubMedGoogle Scholar
  69. 69.
    Robertson DJ, Kaminski MF, Bretthauer M (2015) Effectiveness, training and quality assurance of colonoscopy screening for colorectal cancer. GutGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Interdisciplinary Program for Bioengineering, Graduate SchoolSeoul National UniversitySeoulSouth Korea
  2. 2.Department of GastroenterologySeoul National University Boramae Medical CenterSeoulSouth Korea
  3. 3.Department of Biomedical EngineeringChungnam National University College of MedicineDaejeonSouth Korea
  4. 4.Department of GastroenterologyMediplex Sejong HospitalIncheonSouth Korea
  5. 5.Department of Biomedical EngineeringSeoul National University College of MedicineSeoulSouth Korea
  6. 6.Institute of Medical and Biological EngineeringSeoul National UniversitySeoulSouth Korea

Personalised recommendations