Real-Time Feature Matching for the Accurate Recovery of Augmented-Reality Display in Laparoscopic Videos

  • Gustavo A. Puerto-Souza
  • Alberto Castaño-Bardawil
  • Gian-Luca Mariottini
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7815)


Augmented-Reality (AR) displays increase surgeon’s visual awareness of high-risk surgical targets (e.g., the location of a tumor) by accurately overlaying pre-operative radiological 3-D model onto the intra-operative laparoscopic video. Existing AR systems are not robust to sudden camera motion or prolonged occlusions, which can cause the loss of those anchor points tracked along the video sequence, and thus the loss of the AR display. In this paper, we present a novel AR system, integrated with a novel feature-matching method, to automatically recover the lost augmentation by predicting the image locations of the AR anchor image-points after sudden image changes. Extensive experiments on challenging surgical video data are presented that show the accuracy, speed, and robustness of our designs.


Augmented Reality Anchor Point Feature Match Correct Match Augmented Reality System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Richa, R., Bo, A.P.L., Poignet, P.: Towards robust 3d visual tracking for motion compensation in beating heart surgery. Medical Image Analysis (2010)Google Scholar
  2. 2.
    Stoyanov, D., Mylonas, G.P., Deligianni, F., Darzi, A., Yang, G.Z.: Soft-tissue motion tracking and structure estimation for robotic assisted MIS procedures. In: Duncan, J.S., Gerig, G. (eds.) MICCAI 2005. LNCS, vol. 3750, pp. 139–146. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  3. 3.
    Mountney, P., Yang, G.-Z.: Motion compensated Slam for image guided surgery. In: Jiang, T., Navab, N., Pluim, J.P.W., Viergever, M.A. (eds.) MICCAI 2010, Part II. LNCS, vol. 6362, pp. 496–504. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  4. 4.
    Giannarou, S., Visentini-Scarzanella, M., Yang, G.: Probabilistic tracking of affine-invariant anisotropic regions. IEEE T.PAMI (2012)Google Scholar
  5. 5.
    Yip, M.C., Lowe, D.G., Salcudean, S.E., Rohling, R.N., Nguan, C.Y.: Real-time methods for long-term tissue feature tracking in endoscopic scenes. In: Abolmaesumi, P., Joskowicz, L., Navab, N., Jannin, P. (eds.) IPCAI 2012. LNCS, vol. 7330, pp. 33–43. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  6. 6.
    Puerto-Souza, G.A., Mariottini, G.L.: Hierarchical multi-affine (HMA) algorithm for fast and accurate feature matching in minimally-invasive surgical images. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS (2012)Google Scholar
  7. 7.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comp. Vis. 60(2), 91–110 (2004)CrossRefGoogle Scholar
  8. 8.
    Cho, M., Lee, J., Lee, K.M.: Feature correspondence and deformable object matching via agglomerative correspondence clustering. In: Proc. 9th Int. Conf. Comp. Vis., pp. 1280–1287 (2009)Google Scholar
  9. 9.
    Puerto-Souza, G.A., Adibi, M., Cadeddu, J.A., Mariottini, G.L.: Adaptive multi-affine (AMA) feature-matching algorithm and its application to minimally-invasive surgery images. In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2371–2376 (September 2011)Google Scholar
  10. 10.
    Bay, H., Tuytelaars, T., Van Gool, L.: SURF: Speeded up robust features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006, Part I. LNCS, vol. 3951, pp. 404–417. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  11. 11.
    Morel, J.M., Yu, G.: ASIFT: A new framework for fully affine invariant image comparison. Journal on Imaging Sciences 2(2), 438–469 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  12. 12.
    Szeliski, R.: Computer vision: algorithms and applications. Springer-Verlag New York Inc. (2010)Google Scholar
  13. 13.
    Mikolajczyk, K., Schmid, C.: A performance evaluation of local descriptors. IEEE Trans. Pattern Anal. 60, 1615–1630 (2005)CrossRefGoogle Scholar
  14. 14.
    Mountney, P., Lo, B., Thiemjarus, S., Stoyanov, D., Yang, G.-Z.: A probabilistic framework for tracking deformable soft tissue in minimally invasive surgery. In: Ayache, N., Ourselin, S., Maeder, A. (eds.) MICCAI 2007, Part II. LNCS, vol. 4792, pp. 34–41. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  15. 15.
    Del Bimbo, A., Franco, F., Pernici, F.: Local shape estimation from a single keypoint. In: Proc. Comp. Vis. Patt. Rec. Workshops, pp. 23–28 (2010)Google Scholar
  16. 16.
    Hartley, R., Zisserman, A.: Multiple view geometry in computer vision. Cambridge Univ. Press (2000)Google Scholar
  17. 17.
    Hailin, J., Favaro, P., Soatto, S.: Real-time feature tracking and outlier rejection with changes in illumination. In: Proc. IEEE Intern. Conf. on Computer Vision, vol. 1, pp. 684–689 (2001)Google Scholar
  18. 18.
    Bay, H., Tuytelaars, T., Van Gool, L.: SURF: Speeded up robust features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006, Part I. LNCS, vol. 3951, pp. 404–417. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  19. 19.
    Urban, M., Matas, J., Chum, O., Pajdla, T.: Robust wide baseline stereo for maximally stable extremal regions. Image and Vision Computing 22(10), 761–767 (2004)CrossRefGoogle Scholar
  20. 20.
    Siegwart, R., Nourbakhsh, I.R., Scaramuzza, D.: Introduction to Autonomous Mobile Robots. MIT Press (2011)Google Scholar
  21. 21.
    Rosten, E., Drummond, R.: Fusing points and lines for high performance tracking. In: Proceedings of the International Conference on Computer Vision, pp. 1508–1511 (2005)Google Scholar
  22. 22.
    Harris, C., Stephens, M.: A combined corner and edge detector. In: Alvey Vision Conference, Manchester, UK, vol. 15, p. 50 (1988)Google Scholar
  23. 23.
    Muja, M., Lowe, D.G.: Fast approximate nearest neighbors with automatic algorithm configuration. In: International Conference on Computer Vision Theory and Application, VISSAPP 2009, pp. 331–340. INSTICC Press (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Gustavo A. Puerto-Souza
    • 1
  • Alberto Castaño-Bardawil
    • 1
  • Gian-Luca Mariottini
    • 1
  1. 1.Department of Computer Science and EngineeringUniversity of TexasArlingtonUSA

Personalised recommendations