Real-time image composition of bladder mosaics in fluorescence endoscopy

  • Alexander BehrensEmail author
  • Michael Bommes
  • Thomas Stehle
  • Sebastian Gross
  • Steffen Leonhardt
  • Til Aach
Special Issue Paper


Today, photodynamic diagnostics is commonly used in endoscopic intervention of the urinary bladder. Excited by a narrow band illumination, fluorescence markers enhance the visual contrast between benign and malignant tissue. Since in this modality the endoscope must be moved close to the bladder wall to provide sufficiently exposed images, the field of view (FOV) of the endoscope is very limited. This impedes the navigation and the re-identifying of multi-focal tumors for the physician. Thus, an image providing a larger FOV, composed from single images is highly desired during the intervention for surgery assistance. Since endoscopic mosaicking in real-time is still an open issue, we introduce a new feature-based image mosaicking algorithm for fluorescence endoscopy. Using a multi-threaded software design, the extraction of SURF features, the matching and the image stitching are separated in single processing threads. In an optimization step we discuss the trade-off between feature repeatability and processing time. After adjusting an optimal thread synchronization, the optimal workload of each thread results in a fast and real-time capable computation of image mosaics. On a standard hardware platform our algorithm performs within the RealTimeFrame framework with an update rate of 8.17 frames per second at full input image resolution (780×576). Providing a fast growing image with an extended FOV during the intervention, the mosaic displayed on a second monitor promises high potential for surgery assistance.


Bladder cancer Fluorescence endoscopy Image mosaicking Panorama Image composition 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    American Cancer Society: Cancer facts and figures 2009 Google Scholar
  2. 2.
    Bagley DH, Huffman JL, Lyon ES (1985) Urologic endoscopy: a manual and atlas. Lippincott Williams and Wilkins, Philadelphia Google Scholar
  3. 3.
    Bay H, Ess A, Tuytelaars T, Van Gool L (2008) Surf: Speeded up robust features. Comput Vis Image Underst (CVIU) 110(3):346–359 CrossRefGoogle Scholar
  4. 4.
    Behrens A (2008) Creating panoramic images for bladder fluorescence endoscopy. Acta Polytech J Adv Eng 48(3):50–54 MathSciNetGoogle Scholar
  5. 5.
    Behrens A, Bommes M, Stehle T, Gross S, Leonhardt S, Aach T (2010) Mosaickingalgorithmus zur schnellen Panoramabilderstellung in der Fluoreszenzendoskopie. In: Bildverarbeitung für die Medizin (BVM), vol 574. Springer, Berlin, pp 41–45 Google Scholar
  6. 6.
    Behrens A, Bommes M, Stehle T, Gross S, Leonhardt S, Aach T (2010) A multi-threaded mosaicking algorithm for fast image composition of fluorescence bladder images. In: Wong KH, Miga MI (eds) Proc. SPIE medical imaging 2010: visualization, image-guided procedures, and modeling (San Diego, USA), vol 7625. SPIE, Bellingham, p 76252S Google Scholar
  7. 7.
    Behrens A, Stehle T, Gross S, Aach T (2009) Local and global panoramic imaging for fluorescence bladder endoscopy. In: Engineering in medicine and biology society, EMBC 2009. 31th annu int conf of the IEEE, Minneapolis. IEEE, New York, pp 6990–6993 CrossRefGoogle Scholar
  8. 8.
    Hamadou A Ben, Soussen C, Blondel W, Daul C, Wolf D (2009) Comparative study of image registration techniques for bladder video-endoscopy. In: SPIE European conference on biomedical optics—novel optical instrumentation for biomedical applications IV, vol 7371. SPIE, Bellingham Google Scholar
  9. 9.
    Bergen T, Ruthotto S, Munzenmayer C, Rupp S, Paulus D, Winter C (2009) Feature-based real-time endoscopic mosaicking. In: Proc 6th int symp image and signal processing and analysis (ISPA), pp 695–700 Google Scholar
  10. 10.
    Brown M, Szeliski R, Winder S (2005) Multi-image matching using multi-scale oriented patches. In: Proc IEEE computer society conference on computer vision and pattern recognition (CVPR), vol 1, pp 510–517 Google Scholar
  11. 11.
    Cao CG, Milgram P (2000) Disorientation in minimal access surgery: A case study. In: Proc IEA 2000/HFES 2000 congress, vol 4, pp 169–172 Google Scholar
  12. 12.
    Cheng Z, Devarajan D, Radke RJ (2007) Determining vision graphs for distributed camera networks using feature digests. EURASIP J Appl Signal Process 2007(1):220–220 Google Scholar
  13. 13.
    Fischler MA, Bolles RC (1981) Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395 CrossRefMathSciNetGoogle Scholar
  14. 14.
    Gross S, Behrens A, Stehle T (2009) Rapid development of video processing algorithms with RealTimeFrame. In: Proc biomedica, pp 217–220 Google Scholar
  15. 15.
    Gross S, Stehle T (2008) RealTimeFrame—a real time processing framework for medical video sequences. Acta Polytech J Adv Eng 48(3):15–19 Google Scholar
  16. 16.
    Halim I, Tavakkolizadeh A (2008) Notes: The next surgical revolution? Int J Surg 6(4):273–276 CrossRefGoogle Scholar
  17. 17.
    Hartley R, Kang SB (2007) Parameter-free radial distortion correction with center of distortion estimation. IEEE Trans Pattern Anal Mach Intell 29(8):1309–1321 CrossRefGoogle Scholar
  18. 18.
    Höller K, Penne J, Schneider A, Jahn J, Boronat JG, Wittenberg T, Feußner H, Hornegger J (2009) Endoscopic orientation correction. In: Medical image computing and computer-assisted intervention (MICCAI). LNCS, vol 5761. Springer, Berlin, pp 459–466 Google Scholar
  19. 19.
    Hudson MA, Herr HW (1995) Carcinoma in situ of the bladder. J Urol 153:564–572 Google Scholar
  20. 20.
    Hungerhuber E, Stepp H, Kriegmair M, Stief C, Hofstetter A, Hartmann A, Knuechel R, Karl A, Tritschler S, Zaak D (2007) Seven years’ experience with 5-aminolevulinic acid in detection of transitional cell carcinoma of the bladder. Urology 69(2):260–264 CrossRefGoogle Scholar
  21. 21.
    Kassouf W, Swanson D, Kamat AM, Leibovici D, Siefker-Radtke A, Munsell MF, Grossman HB, Dinney CP (2006) Partial cystectomy for muscle invasive urothelial carcinoma of the bladder: A contemporary review of the m. d. Anderson cancer center experience. J Urol 175(6):2058–2062 CrossRefGoogle Scholar
  22. 22.
    Konen W, Breiderhoff B, Scholz M (2007) Real-time image mosaic for endoscopic video sequences. In: Bildverarbeitung für die Medizin (BVM). Springer, Berlin Google Scholar
  23. 23.
    Kourogi M, Kurata T, Hoshino J, Muraoka Y (1999) Real-time image mosaicing from a video sequence. In: Proc IEEE international conference on image processing (ICIP), vol 4, pp 133–137 Google Scholar
  24. 24.
    Lowe D (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60:91–110 CrossRefGoogle Scholar
  25. 25.
    Mann S, Picard R (1994) Virtual bellows: constructing high quality stills from video. In: Proc IEEE international conference image processing (ICIP), vol 1, pp 363–367 Google Scholar
  26. 26.
    Matouschek E (1987) Urologisch-endoskopische Operationen. Schattauer, Stuttgart Google Scholar
  27. 27.
    Miranda-Luna R, Daul C, Blondel W, Hernandez-Mier Y, Wolf D, Guillemin F (2008) Mosaicing of bladder endoscopic image sequences: Distortion calibration and registration algorithm. IEEE Trans Biomed Eng 55(2):541–553 Google Scholar
  28. 28.
    Miranda-Luna R, Hernandez-Mier Y, Daul C, Blondel W, Wolf D (2004) Mosaicing of medical video-endoscopic images: data quality improvement and algorithm testing. In: 1st int conf electrical and electronics engineering (ICEEE), pp 530–535 Google Scholar
  29. 29.
    Mountney P, Yang GZ (2008) Soft tissue tracking for minimally invasive surgery: Learning local deformation online. In: Medical image computing and computer-assisted intervention (MICCAI). LNCS, vol 5242. Springer, Berlin, pp 364–372 Google Scholar
  30. 30.
    Mountney P, Yang GZ (2009) Dynamic view expansion for minimal invasive surgery using simultaneous localization and mapping. In: Engineering in medicine and biology society, EMBC 2009. 31th annu int conf of the IEEE, pp 1184–1187 Google Scholar
  31. 31.
    Olijnyk S, Mier YH, Blondel WM, Daul C, Wolf D, Bourg-Heckly G (2007) Combination of panoramic and fluorescence endoscopic images to obtain tumor spatial distribution information useful for bladder cancer detection. In: SPIE European conference on biomedical optics—novel optical instrumentation for biomedical applications III, vol 6631. SPIE, Bellingham, p 66310X Google Scholar
  32. 32.
    Orozco RE, Martin AA, Murphy WM (1994) Carcinoma in situ of the urinary bladder clues to host involvement in human carcinogenesis. Cancer 74(1):115–122 CrossRefGoogle Scholar
  33. 33.
    Rattner D, Kalloo A (2006) Asge/sages working group on natural orifice translumenal endoscopic surgery. Surg Endosc 20(2):329–333 CrossRefGoogle Scholar
  34. 34.
    Seshamani S, Lau WW, Hager GD (2006) Real-time endoscopic mosaicking. In: Medical image computing and computer-assisted intervention (MICCAI). LNCS, vol 4190. Springer, Berlin, pp 355–363 Google Scholar
  35. 35.
    Stehle T, Behrens A, Aach T (2008) Enhancement of visual contrast in fluorescence endoscopy. In: IEEE international conference on multimedia and expo Google Scholar
  36. 36.
    Stehle T, Behrens A, Aach T (2008) Visual enhancement of facial tissue in endoscopy. In: SPIE medical imaging 2008: visualization, image guided procedures, and modeling, vol 6918. SPIE, Bellingham Google Scholar
  37. 37.
    Stehle T, Wulff J, Behrens A, Gross S, Aach T (2009) Denoising fluorescence endoscopy: A motion-compensated temporal recursive video filter with an optimal minimum mean square error parametrization. In: Proceedings of IEEE international symposium on biomedical imaging (ISBI), pp 314–317 Google Scholar
  38. 38.
    Szeliski R (1996) Video mosaics for virtual environments. IEEE Comput Graph Appl 16(2):22–30 CrossRefGoogle Scholar
  39. 39.
    Szeliski R (2006) Image alignment and stitching: A tutorial. Tech rep MSR-TR-2004-92, Microsoft Research Google Scholar
  40. 40.
    Thormählen T, Broszio H, Meier PN (2002) Three-dimensional endoscopy. In: Falk symposium, medical imaging in gastroenterology and hepatology. Kluwer Academic, Dordrecht, p 124 Google Scholar
  41. 41.
    Tomasi C, Kanade T (1991) Detection and tracking of point features. Tech rep CMU-CS-91-132. Carnegie Mellon University Google Scholar
  42. 42.
    Triggs B, Mclauchlan PF, Hartley RI, Fitzgibbon AW (2000) Bundle adjustment—a modern synthesis. In: Vision algorithms: theory and practice. LCNS, vol 1883. Springer, Berlin, pp 298–375 CrossRefGoogle Scholar
  43. 43.
    Wald D, Reeff M, Székely G, Cattin P, Paulus D (2005) Fließende Überblendung von Endoskopiebildern für die Erstellung eines Mosaiks. In: Bildverarbeitung für die Medizin (BVM). Springer, Berlin, pp 287–291 CrossRefGoogle Scholar
  44. 44.
    Wickham JEA (1987) The new surgery. Br Med J 295(6613):1581–1582 CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2010

Authors and Affiliations

  • Alexander Behrens
    • 1
    Email author
  • Michael Bommes
    • 1
  • Thomas Stehle
    • 1
  • Sebastian Gross
    • 1
  • Steffen Leonhardt
    • 2
  • Til Aach
    • 1
  1. 1.Institute of Imaging & Computer VisionRWTH Aachen UniversityAachenGermany
  2. 2.Philips Chair for Medical Information Technology, Helmholtz-InstituteRWTH Aachen UniversityAachenGermany

Personalised recommendations