Journal of Medical Systems

, 42:138 | Cite as

An Augmented Reality Endoscope System for Ureter Position Detection

  • Feng Yu
  • Enmin Song
  • Hong LiuEmail author
  • Yunlong Li
  • Jun Zhu
  • Chih-Cheng Hung
Image & Signal Processing
Part of the following topical collections:
  1. Advanced Computational Intelligence and Soft Computing in Medical Imaging


Iatrogenic injury of ureter in the clinical operation may cause the serious complication and kidney damage. To avoid such a medical accident, it is necessary to provide the ureter position information to the doctor. For the detection of ureter position, an ureter position detection and display system with the augmented ris proposed to detect the ureter that is covered by human tissue. There are two key issues which should be considered in this new system. One is how to detect the covered ureter that cannot be captured by the electronic endoscope and the other is how to display the ureter position that provides stable and high-quality images. Simultaneously, any delayed processing of the system should disturb the surgery. The aided hardware detection method and target detection algorithms are proposed in this system. To mark the ureter position, a surface-lighting plastic optical fiber (POF) with the encoded light-emitting diode (LED) light is used to indicate the ureter position. The monochrome channel filtering algorithm (MCFA) is proposed to locate the ureter region more precisely. The ureter position is extracted using the proposed automatic region growing algorithm (ARGA) that utilizes the statistical information of the monochrome channel for the selection of growing seed point. In addition, according to the pulse signal of encoded light, the recognition of bright and dark frames based on the aided hardware (BDAH) is proposed to expedite the processing speed. Experimental results demonstrate that the proposed endoscope system can identify 92.04% ureter region in average.


Ureter injury Endoscope system Ureter position detection Augmented reality Automatic region growing algorithm (ARGA



This work was supported by National Key R & D Program of China, No. 2017YFC0112804, National Natural Science Foundation of China under grant project No.61370179, the Fundamental Research Funds for the Central Universities, HUST: 2016YXZD018 and HUST: 2017JYCX038, Medical Clinical Science and Technology Development Fund of Jiangsu University, No. JLY20140051C, and Clinical Medicine Science and Technology Projects in Jiangsu province, No. BL2014056.

Compliance with Ethical Standards

Conflict of interests

The authors declare no conflict of interest.

This article does not contain any studies with human participants or animals performed by any of the authors.


  1. 1.
    Ševo, I., Avramović, A., Balasingham, I., Elle, O., Bergsland, J., and Aabakken, L., Edge density based automatic detection of inflammation in colonoscopy videos. Comput. Biol. Med. 72:138–150, 2016.CrossRefPubMedGoogle Scholar
  2. 2.
    Reynisson, P. J., Leira, H. O., Hernes, T. N., Hofstad, E. F., Scali, M., Sorger, H., Amundsen, T., Lindseth, F., and Langø, T., Navigated bronchoscopy: a technical review. Journal of Bronchology & Interventional Pulmonology 21(3):242–264, 2014.CrossRefGoogle Scholar
  3. 3.
    Lambert, R., Prevention of gastrointestinal cancer by surveillance endoscopy. EPMA J. 1(3):473–483, 2010.CrossRefPubMedPubMedCentralGoogle Scholar
  4. 4.
    Lurie, K. L., Smith, G. T., Khan, S. A., Liao, J. C., and Ellerbee, A. K., Three-dimensional, distendable bladder phantom for optical coherence tomography and white light cystoscopy. J. Biomed. Opt. 19(3):036 009-036 009, 2014.CrossRefGoogle Scholar
  5. 5.
    Selka, F., Nicolau, S., Agnus, V., Bessaid, A., Marescaux, J., and Soler, L., Context-specific selection of algorithms for recursive feature tracking in endoscopic image using a new methodology. Comput. Med. Imaging Graph. 40:49–61, 2015.CrossRefPubMedGoogle Scholar
  6. 6.
    Oh, S. Y., Kwon, S., Lee, K. G., Suh, Y. S., Choe, H. N., Kong, S. H., Lee, H. J., Kim, W. H., and Yang, H. K., Outcomes of minimally invasive surgery for early gastric cancer are comparable with those for open surgery: analysis of 1,013 minimally invasive surgeries at a single institution. Surg. Endosc. 28(3):789–795, 2014.CrossRefPubMedGoogle Scholar
  7. 7.
    Burdall, O. C., Boddy, A. P., Fullick, J., Blazeby, J., Krysztopik, R., Streets, C., Hollowood, A., Barham, C. P., and Titcomb, D., A comparative study of survival after minimally invasive and open oesophagectomy. Surg. Endosc. 29(2):431–437, 2015.CrossRefPubMedGoogle Scholar
  8. 8.
    Clayman, R. V., Kavoussi, L. R., Soper, N. J., Dierks, S. M., Meretyk, S., Darcy, M. D., Roemer, F. D., Pingleton, E. D., Thomson, P. G., and Long, S. R., Laparoscopic nephrectomy: initial case report. J. Urol. 197(2):S182–S186, 2017.CrossRefPubMedGoogle Scholar
  9. 9.
    Martin-Perez, B., Andrade-Ribeiro, G., Hunter, L., and Atallah, S., A systematic review of transanal minimally invasive surgery (tamis) from 2010 to 2013. Tech. Coloproctol. 18(9):775–788, 2014.CrossRefPubMedGoogle Scholar
  10. 10.
    Van den Haak, L., Alleblas, C., Nieboer, T., Rhemrev, J., and Jansen, F., Efficacy and safety of uterine manipulators in laparoscopic surgery: a review. Arch. Gynecol. Obstet. 292(5):1003–1011, 2015.CrossRefGoogle Scholar
  11. 11.
    Aronson, M. P., and Bose, T. M., Urinary tract injury in pelvic surgery. Clin. Obstet. Gynecol. 45(2):428–438, 2002.CrossRefPubMedGoogle Scholar
  12. 12.
    Harkki-Siren, P., Sjoberg, J., and Tiitinen, A., Urinary tract injuries after hysterectomy. Obstet. Gynecol. 92(1):113–118, 1998.CrossRefPubMedGoogle Scholar
  13. 13.
    Parpala-Spårman, T., Paananen, I., Santala, M., Ohtonen, P., and Hellström, P., Increasing numbers of ureteric injuries after the introduction of laparoscopic surgery. Scand. J. Urol. Nephrol. 42(5):422–427, 2008.CrossRefPubMedGoogle Scholar
  14. 14.
    Orr, W. S., Pisters, L. L., and Rodriguez-Bigas, M.A.: Intraoperative ureteral injury 34. Gastrointestinal Surgery: Management of Complex Perioperative Complications, p 361, 2015Google Scholar
  15. 15.
    Acher, C., and Agarwal, S.: Injury of the kidney, ureter, and bladder. In: Penetrating trauma, Springer, pp. 387–396, 2017.Google Scholar
  16. 16.
    Wu, C. J., Tseng, C. W., and Wu, M. P., Laparoscopic subtotal hysterectomy in the era of minimally invasive surgery. Gynecology and Minimally Invasive Therapy 4(1):8–13, 2015.CrossRefGoogle Scholar
  17. 17.
    Janssen, P. F., Brölmann, H. A., and Huirne, J. A., Causes and prevention of laparoscopic ureter injuries: an analysis of 31 cases during laparoscopic hysterectomy in the netherlands. Surg. Endosc. 27(3):946–956, 2013.CrossRefPubMedGoogle Scholar
  18. 18.
    Cuesta, M. A.: Case on ureter lesion during laparoscopic low anterior resection. In: Case Studies of Postoperative Complications after Digestive Surgery, Springer, pp. 407–413, 2014.Google Scholar
  19. 19.
    Schimpf, M., and Gottenger, E., Universal ureteral stent placement at hysterectomy to identify ureteral injury: a decision analysis. J. Wagner, BJOG: An International Journal of Obstetrics & Gynaecology 115(9):1151–1158, 2008.CrossRefGoogle Scholar
  20. 20.
    Fu, W. J., Wang, Z. X., Li, G., Cui, F. Z., Zhang, Y., and Zhang, X., Comparison of a biodegradable ureteral stent versus the traditional double-j stent for the treatment of ureteral injury: an experimental study. Biomed. Mater. 7(6):065002, 2012.CrossRefPubMedGoogle Scholar
  21. 21.
    Chahin, F., Dwivedi, A. J., Paramesh, A., Chau, W., Agrawal, S., Chahin, C., Kumar, A., Tootla, A., Tootla, F., and Silva, Y. J., The implications of lighted ureteral stenting in laparoscopic colectomy. Jsls 6(1):49–52, 2002.PubMedPubMedCentralGoogle Scholar
  22. 22.
    Kwon, I. G., Cho, I., Guner, A., Choi, Y. Y., Shin, H. B., Kim, H. I., An, J. Y., Cheong, J. H., Noh, S. H., and Hyung, W. J., Minimally invasive surgery for remnant gastric cancer: a comparison with open surgery. Surg. Endosc. 28(8):2452–2458, 2014.CrossRefPubMedGoogle Scholar
  23. 23.
    Uttley, L., Campbell, F., Rhodes, M., Cantrell, A., Stegenga, H., and Lloyd-Jones, M., Minimally invasive oesophagectomy versus open surgery: is there an advantage? Surg. Endosc. 27(3):724–731, 2013.CrossRefPubMedGoogle Scholar
  24. 24.
    Tanaka, E., Ohnishi, S., Laurence, R. G., Choi, H. S., Humblet, V., and Frangioni, J. V., Real-time intraoperative ureteral guidance using invisible near-infrared fluorescence. J. Urol. 178(5):2197–2202, 2007.CrossRefPubMedPubMedCentralGoogle Scholar
  25. 25.
    Verbeek, F. P. R., Vorst, J. R. V. D., Schaafsma, B. E., Swijnenburg, R. J., Gaarenstroom, K. N., Elzevier, H. W., Velde, C. J. H. V. D., Frangioni, J. V., and Vahrmeijer, A. L., Intraoperative near infrared fluorescence guided identification of the ureters using low dose methylene blue: A first in human experience. J. Urol. 190(2):574–579, 2013.CrossRefPubMedPubMedCentralGoogle Scholar
  26. 26.
    Al-Taher, M., van den Bos, J., Schols, R. M., Bouvy, N. D., and Stassen, L. P., Fluorescence ureteral visualization in human laparoscopic colorectal surgery using methylene blue. J. Laparoendosc. Adv. Surg. Tech. 26(11):870–875, 2016.CrossRefGoogle Scholar
  27. 27.
    Song, E., Yu, F., Liu, H., Cheng, N., Li, Y., Jin, L., and Hung, C. C., A novel endoscope system for position detection and depth estimation of the ureter. J. Med. Syst. 40(12):266, 2016.CrossRefPubMedGoogle Scholar
  28. 28.
    Loukas, C., Lahanas, V., and Georgiou, E., An integrated approach to endoscopic instrument tracking for augmented reality applications in surgical simulation training. Int. J. Med. Rob. Comput. Assisted Surg 9(4):e34–e51, 2013.CrossRefGoogle Scholar
  29. 29.
    Yamamoto, T., Abolhassani, N., Jung, S., Okamura, A. M., and Judkins, T. N., Augmented reality and haptic interfaces for robot-assisted surgery. Int. J. Med. Rob. Comput. Assisted Surg. 8(1):45–56, 2012.CrossRefGoogle Scholar
  30. 30.
    Pagador, J. B., Sánchez, L., Sánchez, J., Bustos, P., Moreno, J., and Sánchez-margallo, F. M., Augmented reality haptic (arh): an approach of electromagnetic tracking in minimally invasive surgery. Int. J. Comput. Assist. Radiol. Surg. 6(2):257–263 , 2011.CrossRefPubMedGoogle Scholar
  31. 31.
    Bernhardt, S., Nicolau, S. A., Agnus, V., Soler, L., Doignon, C., and Marescaux, J., Automatic localization of endoscope in intraoperative ct image: a simple approach to augmented reality guidance in laparoscopic surgery. Med. Image Anal. 30:130–143, 2016.CrossRefPubMedGoogle Scholar
  32. 32.
    Katić, D., Wekerle, A. L., Görtler, J., Spengler, P., Bodenstedt, S., Röhl, S., Suwelack, S., Kenngott, H. G., Wagner, M., Müller-Stich, B. P., et al., Context-aware augmented reality in laparoscopic surgery. Comput. Med. Imaging Graph. 37(2):174–182, 2013.CrossRefPubMedGoogle Scholar
  33. 33.
    Puerto-Souza, G. A., Cadeddu, J. A., and Mariottini, G. L., Toward long-term and accurate augmented-reality for monocular endoscopic videos. IEEE Trans. Biomed. Eng. 61(10):2609–2620, 2014.CrossRefPubMedGoogle Scholar
  34. 34.
    Russ, J. C., Matey, J. R., Mallinckrodt, A. J., and Mckay, S., The image processing handbook. Comput. Phys. 8(2):177, 2002.CrossRefGoogle Scholar
  35. 35.
    Good, M. M., Abele, T. A., Balgobin, S., Montoya, T. I., McIntire, D., and Corton, M. M., Vascular and ureteral anatomy relative to the midsacral promontory. Am. J. Obstet. Gynecol. 208(6):486–e1, 2013.CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of Computer and TechnologyHuazhong University of Science and TechnologyWuhanChina
  2. 2.Department of UrologyAffiliated Kunsan Hospital of Jiangsu UniversityKunshanChina
  3. 3.Center for Machine Vision and Security ResearchKennesaw State UniversityKennesawUSA

Personalised recommendations