Soft tissue motion tracking with application to tablet-based incision planning in laser surgery

International Journal of Computer Assisted Radiology and Surgery volume 11, pages2325–2337(2016)Cite this article

Abstract

Purpose

Recent research has revealed that incision planning in laser surgery deploying stylus and tablet outperforms micromanipulator control. However, vision-based adaption to dynamic surgical scenes has not been addressed so far. In this study, scene motion compensation for tablet-based planning by means of tissue deformation tracking is discussed.

Methods

A stereo-based method for motion tracking with piecewise affine deformation modeling is presented. Proposed parametrization relies on the epipolar constraint to enforce left–right consistency in the energy minimization problem. Furthermore, the method implements illumination-invariant tracking and appearance-based occlusion detection. Performance is assessed on laparoscopic and laryngeal in vivo data. In particular, tracking accuracy is measured under various conditions such as occlusions and simulated laser cuttings. Experimental validation is extended to a user study conducted on a tablet-based interface that integrates the tracking for image stabilization.

Results

Tracking accuracy measurement reveals a root-mean-square error of 2.45 mm for the laparoscopic and 0.41 mm for the laryngeal dataset. Results successfully demonstrate stereoscopic tracking under changes in illumination, translation, rotation and scale. In particular, proposed occlusion detection scheme can increase robustness against tracking failure. Moreover, assessed user performance indicates significantly increased path tracing accuracy and usability if proposed tracking is deployed to stabilize the view during free-hand path definition.

Conclusion

The presented algorithm successfully extends piecewise affine deformation tracking to stereo vision taking the epipolar constraint into account. Improved surgical performance as demonstrated for laser incision planning highlights the potential of presented method regarding further applications in computer-assisted surgery.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

US$ 39.95

Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Notes

  1. 1.

    http://hamlyn.doc.ic.ac.uk/vision/.

  2. 2.

    http://www.microralp.eu/.

References

  1. 1.

    Andreff N, Tamadazte B (2016) Laser steering using virtual trifocal visual servoing. Int J Robot Res 35(6):672–694. doi:10.1177/0278364915585585

    Article  Google Scholar 

  2. 2.

    Bouguet JY (2000) Pyramidal implementation of the lucas kanade feature tracker. Microprocessor Research Labs, Intel Corporation, Tech. rep

  3. 3.

    Brunet F, Gay-Bellile V, Bartoli A, Navab N, Malgouyres R (2011) Feature-driven direct non-rigid image registration. Int J Comput Vis 93(1):33–52. doi:10.1007/s11263-010-0407-x

    Article  Google Scholar 

  4. 4.

    Dagnino G, Mattos L, Caldwell D (2015) A vision-based system for fast and accurate laser scanning in robot-assisted phonomicrosurgery. Int J Comput Assist Radiol Surg 10(2):217–229. doi:10.1007/s11548-014-1078-9

    Article  PubMed  Google Scholar 

  5. 5.

    Du X, Clancy N, Arya S, Hanna G, Kelly J, Elson D, Stoyanov D (2015) Robust surface tracking combining features, intensity and illumination compensation. Int J Comput Assist Radiol Surg 10(12):1915–1926. doi:10.1007/s11548-015-1243-9

    Article  PubMed  Google Scholar 

  6. 6.

    Fichera L, Pardo D, Illiano P, Ortiz J, Caldwell DG, Mattos LS (2016) Online estimation of laser incision depth for transoral microsurgery: approach and preliminary evaluation. Int J Med Robot Comput Assist Surg 12(1):53–61. doi:10.1002/rcs.1656

    Article  Google Scholar 

  7. 7.

    Filzmoser P, Ruiz-Gazen A, Thomas-Agnan C (2013) Identification of local multivariate outliers. Stat Pap 55(1):29–47. doi:10.1007/s00362-013-0524-z

    Article  Google Scholar 

  8. 8.

    Giannarou S, Visentini-Scarzanella M, Yang GZ (2013) Probabilistic tracking of affine-invariant anisotropic regions. Pattern Anal Mach Intell IEEE Trans 35(1):130–143. doi:10.1109/TPAMI.2012.81

    Article  Google Scholar 

  9. 9.

    Kalal Z, Mikolajczyk K, Matas J (2010) Forward-backward error: automatic detection of tracking failures. In: Pattern recognition (ICPR), 2010 20th international conference on, pp 2756–2759. doi:10.1109/ICPR.2010.675

  10. 10.

    Lau W, Ramey N, Corso J, Thakor N, Hager G (2004) Stereo-based endoscopic tracking of cardiac surface deformation. In: Medical image computing and computer-assisted intervention, MICCAI 2004, vol 3217, pp 494–501. doi:10.1007/978-3-540-30136-3_61

  11. 11.

    Lewis JR (1991) Psychometric evaluation of an after-scenario questionnaire for computer usability studies: the ASQ. ACM SIGCHI Bull 23(1):78–81. doi:10.1145/122672.122692

    Article  Google Scholar 

  12. 12.

    Mattos LS, Deshpande N, Barresi G, Guastini L, Peretti G (2014) A novel computerized surgeon-machine interface for robot-assisted laser phonomicrosurgery. Laryngoscope 124(8):1887–1894. doi:10.1002/lary.24566

    Article  PubMed  Google Scholar 

  13. 13.

    McGill R, Tukey JW, Larsen WA (1978) Variations of box plots. Am Stat 32(1):12–16

    Google Scholar 

  14. 14.

    Mountney P, Stoyanov D, Yang GZ (2010) Three-dimensional tissue deformation recovery and tracking. Signal Process Mag IEEE 27(4):14–24. doi:10.1109/MSP.2010.936728

    Article  Google Scholar 

  15. 15.

    Mountney P, Yang GZ (2008) Soft tissue tracking for minimally invasive surgery: learning local deformation online. Med Image Comput Comput Assist Interv MICCAI 5242:364–372. doi:10.1007/978-3-540-85990-1_44

    Google Scholar 

  16. 16.

    Niemz MH (2013) Laser-tissue interactions: fundamentals and applications. Springer Science & Business Media, Berlin

    Google Scholar 

  17. 17.

    Ortmaier T, Gröger M, Boehm DH, Falk V, Hirzinger G (2005) Motion estimation in beating heart surgery. IEEE Trans Biomed Eng 52(10):1729–1740. doi:10.1109/TBME.2005.855716

    Article  PubMed  Google Scholar 

  18. 18.

    Pilet J, Lepetit V, Fua P (2008) Fast non-rigid surface detection, registration and realistic augmentation. Int J Comput Vis 76(2):109–122. doi:10.1007/s11263-006-0017-9

    Article  Google Scholar 

  19. 19.

    Puerto-Souza G, Mariottini GL (2013) A fast and accurate feature-matching algorithm for minimally-invasive endoscopic images. Med Imaging IEEE Trans 32(7):1201–1214. doi:10.1109/TMI.2013.2239306

    Article  Google Scholar 

  20. 20.

    Richa R, Poignet P, Liu C (2010) Three-dimensional motion tracking for beating heart surgery using a thin-plate spline deformable model. Int J Rob Res 29(2–3):218–230. doi:10.1177/0278364909356600

    Article  Google Scholar 

  21. 21.

    Rubinstein M, Armstrong W (2011) Transoral laser microsurgery for laryngeal cancer: a primer and review of laser dosimetry. Lasers Med Sci 26(1):113–124. doi:10.1007/s10103-010-0834-5

    Article  PubMed  Google Scholar 

  22. 22.

    Sauvée M, Poignet P, Triboulet J, Dombre E, Malis E, Demaria R (2006) 3d heart motion estimation using endoscopic monocular vision system. Model Control Biomed Syst 6:141–146. doi:10.3182/20060920-3-FR-2912.00029

    Google Scholar 

  23. 23.

    Schoob A, Kundrat D, Kahrs L, Ortmaier T (2016) Comparative study on surface reconstruction accuracy of stereo imaging devices for microsurgery. Int J Comput Assist Radiol Surg 11(1):145–156. doi:10.1007/s11548-015-1240-z

    Article  PubMed  Google Scholar 

  24. 24.

    Schoob A, Kundrat D, Kleingrothe L, Kahrs L, Andreff N, Ortmaier T (2015) Tissue surface information for intraoperative incision planning and focus adjustment in laser surgery. Int J Comput Assist Radiol Surg 10(2):171–181. doi:10.1007/s11548-014-1077-x

    Article  PubMed  Google Scholar 

  25. 25.

    Schoob A, Lekon S, Kundrat D, Kahrs LA, Mattos LS, Ortmaier T (2015) Comparison of tablet-based strategies for incision planning in laser microsurgery. In: Proceedings of SPIE medical imaging: image-guided procedures, vol 9415. doi:10.1117/12.2081032

  26. 26.

    Sotiras A, Davatzikos C, Paragios N (2013) Deformable medical image registration: a survey. Med Imaging IEEE Trans 32(7):1153–1190. doi:10.1109/TMI.2013.2265603

    Article  Google Scholar 

  27. 27.

    Stoyanov D, Darzi A, Yang GZ (2005) A practical approach towards accurate dense 3d depth recovery for robotic laparoscopic surgery. Comput Aided Surg 10(4):199–208. doi:10.3109/10929080500230379

    Article  PubMed  Google Scholar 

  28. 28.

    Stoyanov D, Mylonas GP, Deligianni F, Darzi A, Yang GZ (2005) Soft-tissue motion tracking and structure estimation for robotic assisted MIS procedures. In: Medical image computing and computer-assisted intervention—MICCAI 2005, pp 139–146. Springer, Berlin. doi:10.1007/11566489_18

  29. 29.

    Stoyanov D, Yang GZ (2007) Stabilization of image motion for robotic assisted beating heart surgery. In: Medical image computing and computer-assisted intervention, MICCAI 2007, vol 4791, pp 417–424 . doi:10.1007/978-3-540-75757-3_51

  30. 30.

    Stoyanov D, Yang GZ (2009) Soft tissue deformation tracking for robotic assisted minimally invasive surgery. In: Engineering in medicine and biology society, 2009. EMBC 2009. Annual international conference of the IEEE, pp 254–257. doi:10.1109/IEMBS.2009.5334010

  31. 31.

    Tan DJ, Holzer S, Navab N, Ilic S (2014) Deformable template tracking in 1ms. In: Proceedings of the British machine vision conference. BMVA Press

  32. 32.

    Tang HW, Brussel HV, Sloten JV, Reynaerts D, De Win G, Cleynenbreugel BV, Koninckx PR (2006) Evaluation of an intuitive writing interface in robot-aided laser laparoscopic surgery. Comput Aided Surg 11(1):21–30. doi:10.3109/10929080500450886

    Article  PubMed  Google Scholar 

  33. 33.

    Visentini-Scarzanella M, Merrifield R, Stoyanov D, Yang GZ (2010)Tracking of irregular graphical structures for tissue deformation recovery in minimally invasive surgery. In: Jiang T, Navab N, Pluim J, Viergever M (eds.) Medical image computing and computer-assisted intervention, MICCAI 2010, vol 6363. Springer, Heidelberg, pp 261–268 . doi:10.1007/978-3-642-15711-0_33

  34. 34.

    Wong WK, Yang B, Liu C, Poignet P (2013) A quasi-spherical triangle-based approach for efficient 3-d soft-tissue motion tracking. Mechatron IEEE/ASME Trans 18(5):1472–1484. doi:10.1109/TMECH.2012.2203919

    Article  Google Scholar 

  35. 35.

    Yang B, Wong WK, Liu C, Poignet P (2014) 3d soft-tissue tracking using spatial-color joint probability distribution and thin-plate spline model. Pattern Recognit 47(9):2962–2973. doi:10.1016/j.patcog.2014.03.020

    Article  Google Scholar 

  36. 36.

    Ye M, Giannarou S, Meining A, Yang GZ (2016) Online tracking and retargeting with applications to optical biopsy in gastrointestinal endoscopic examinations. Med Image Anal 30:144–157. doi:10.1016/j.media.2015.10.003

    Article  PubMed  Google Scholar 

  37. 37.

    Yip M, Lowe D, Salcudean S, Rohling R, Nguan C (2012) Tissue tracking and registration for image-guided surgery. Med Imaging IEEE Trans 31(11):2169–2182. doi:10.1109/TMI.2012.2212718

    Article  Google Scholar 

  38. 38.

    Zabih R, Woodfill J (1994) Non-parametric local transforms for computing visual correspondence. In: Eklundh JO (ed) Computer vision ECCV 94, vol 801, pp 151–158. Springer, Berlin. doi:10.1007/BFb0028345

  39. 39.

    Zagorchev L, Goshtasby A (2006) A comparative study of transformation functions for nonrigid image registration. Image Process IEEE Trans 15(3):529–538. doi:10.1109/TIP.2005.863114

    Article  Google Scholar 

  40. 40.

    Zhang TY, Suen CY (1984) A fast parallel algorithm for thinning digital patterns. Commun ACM 27(3):236–239. doi:10.1145/357994.358023

    Article  Google Scholar 

  41. 41.

    Zhu J, Lyu M, Huang T (2009) A fast 2d shape recovery approach by fusing features and appearance. Pattern Anal Mach Intell IEEE Trans 31(7):1210–1224. doi:10.1109/TPAMI.2008.151

    Article  Google Scholar 

Download references

Acknowledgments

This research has received funding from the European Union Seventh Framework Programme FP7/2007-2013–Challenge 2—Cognitive Systems, Interaction, Robotics—under grant agreement \(\upmu \)RALP—no 288663. We thank Giorgio Peretti from the Department of Otorhinolaryngology, University of Genoa, Italy, providing in vivo laryngeal data. We also would like to thank Guang-Zhong Yang, Danail Stoyanov and Peter Mountney for the in vivo data provided by the Hamlyn Centre for Robotic Surgery, Imperial College London, UK.

Author information

Affiliations

  1. Institute of Mechatronic Systems, Leibniz Universität Hannover, 30167, Hanover, Germany

    Andreas Schoob, Max-Heinrich Laves, Lüder Alexander Kahrs & Tobias Ortmaier

Authors
  1. Andreas Schoob
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Max-Heinrich Laves
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lüder Alexander Kahrs
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tobias Ortmaier
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andreas Schoob.

Ethics declarations

Conflict of interest

All authors declare that they have no conflict of interest.

Electronic supplementary material

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Schoob, A., Laves, MH., Kahrs, L.A. et al. Soft tissue motion tracking with application to tablet-based incision planning in laser surgery. Int J CARS 11, 2325–2337 (2016). https://doi.org/10.1007/s11548-016-1420-5

Download citation

Keywords

  • Soft tissue deformation tracking
  • Stereo vision
  • Epipolar constraint
  • Surgeon interface
  • User performance