Non-linear-Optimization Using SQP for 3D Deformable Prostate Model Pose Estimation in Minimally Invasive Surgery

  • Daniele Amparore
  • Enrico Checcucci
  • Marco Gribaudo
  • Pietro PiazzollaEmail author
  • Francesco Porpiglia
  • Enrico Vezzetti
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 943)


Augmented Reality began to be used in the last decade to guide and assist the surgeon during minimally invasive surgery. In many AR-based surgical navigation systems, a patient-specific 3D model of the surgical procedure target organ is generated from preoperative images and overlaid on the real views of the surgical field. We are currently developing an AR-based navigation system to support robot-assisted radical prostatectomy (AR-RARP) and in this paper we address the registration and localization challenge of the 3D prostate model during the procedure, evaluating the performances of a Successive Quadratic Programming (SQP) non-linear optimization technique used to align the coordinates of a deformable 3D model to those of the surgical environment. We compared SQP results in solving the 3D pose problem with those provided by the Matlab Computer Vision Toolkit perspective-three-point algorithm, highlighting the differences between the two approaches.


Augmented Reality Robotic surgical procedures Prostatectomy Computed-assisted surgery Successive Quadratic Programming Performance evaluation 


  1. 1.
    Bernhardt, S., Nicolau, S.A., Soler, L., Doignon, C.: The status of augmented reality in laparoscopic surgery as of 2016. Med. Image Anal. 37, 66–90 (2017)CrossRefGoogle Scholar
  2. 2.
    Blender Online Community: Blender (2017).
  3. 3.
    Bradski, G.: The OpenCV library. Dr. Dobb’s J. Softw. Tools 25(11) 120, 122–125 (2000)Google Scholar
  4. 4.
    Cutolo, F.: Augmented Reality in Image-Guided Surgery, pp. 1–11. Springer, Cham (2017)Google Scholar
  5. 5.
    Fida, B., Cutolo, F., di Franco, G., Ferrari, M., Ferrari, V.: Augmented reality in open surgery. Updates Surg 70, 389–400 (2018)CrossRefGoogle Scholar
  6. 6.
    Gao, X.S., Hou, X.R., Tang, J., Cheng, H.F.: Complete solution classification for the perspective-three-point problem. IEEE Trans. Pattern Anal. Mach. Intell. 25(8), 930–943 (2003)CrossRefGoogle Scholar
  7. 7.
    Intuitive: da Vinci Surgical Systems. Accessed 30 Aug 2018
  8. 8.
    Kersten-Oertel, M., Jannin, P., Collins, D.L.: DVV: a taxonomy for mixed reality visualization in image guided surgery. IEEE Trans. Vis. Comput. Graph. 18(2), 332–352 (2012)CrossRefGoogle Scholar
  9. 9.
    Kong, S.H., Haouchine, N., Soares, R., Klymchenko, A., Andreiuk, B., Marques, B., Shabat, G., Piechaud, T., Diana, M., Cotin, S., Marescaux, J.: Robust augmented reality registration method for localization of solid organs’ tumors using CT-derived virtual biomechanical model and fluorescent fiducials. Surg. Endosc. 31(7), 2863–2871 (2017)CrossRefGoogle Scholar
  10. 10.
    Lepetit, V., Moreno-Noguer, F., Fua, P.: EPnP: an accurate O(n) solution to the PnP problem. Int. J. Comput. Vis. 81(2), 155 (2008)CrossRefGoogle Scholar
  11. 11.
    MATLAB: version 8.6.0 (r2015b) (2015)Google Scholar
  12. 12.
    Microsoft Inc.: C# Language Specification (2018). Accessed 30 Aug 2018
  13. 13.
    Nguyen, T.T., Jung, H., Lee, D.Y.: Markerless tracking for augmented reality for image-guided endoscopic retrograde cholangiopancreatography. In: 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 7364–7367, July 2013Google Scholar
  14. 14.
    O’Gorman, F., Clowes, M.B.: Finding picture edges through collinearity of feature points. IEEE Trans. Comput. 25(4), 449–456 (1976)CrossRefGoogle Scholar
  15. 15.
    Porpiglia, F., Bertolo, R., Amparore, D., Checcucci, E., Artibani, W., Dasgupta, P., Montorsi, F., Tewari, A., Fiori, C.: Augmented reality during robot-assisted radical prostatectomy: expert robotic surgeons’ on-the-spot insights after live surgery. Minerva Urologica e Nefrologica 70(2), 226–229 (2018)Google Scholar
  16. 16.
    Porpiglia, F., Bertolo, R., Checcucci, E., Amparore, D., Autorino, R., Dasgupta, P., Wiklund, P., Tewari, A., Liatsikos, E., Fiori, C.: The ESUT research group: development and validation of 3D printed virtual models for robot-assisted radical prostatectomy and partial nephrectomy: urologists’ and patients’ perception. World J. Urol. 36(2), 201–207 (2018)CrossRefGoogle Scholar
  17. 17.
    Porpiglia, F., Checcucci, E., Amparore, D., Autorino, R., Piana, A., Bellin, A., Piazzolla, P., Massa, F., Bollito, E., Gned, D., De Pascale, A., Fiori, C.: Augmented-reality robot-assisted radical prostatectomy using hyper-accuracy three-dimensional reconstruction (HA3D\({^{\rm TM}}\)) technology: a radiological and pathological study. BJU Int. (2018)Google Scholar
  18. 18.
    Porpiglia, F., Fiori, C., Checcucci, E., Amparore, D., Bertolo, R.: Augmented reality robot-assisted radical prostatectomy: preliminary experience. Urology 115, 184 (2018)CrossRefGoogle Scholar
  19. 19.
    Sederberg, T.W., Parry, S.R.: Free-form deformation of solid geometric models. SIGGRAPH Comput. Graph. 20(4), 151–160 (1986)CrossRefGoogle Scholar
  20. 20.
    Simpfendörfer, T., Baumhauer, M., Müller, M., Gutt, C.N., Meinzer, H.P., Rassweiler, J.J., Guven, S., Teber, D.: Augmented reality visualization during laparoscopic radical prostatectomy. J. Endourol. 25, 1841–1845 (2011)CrossRefGoogle Scholar
  21. 21.
    Thompson, S., Schneider, C., Bosi, M., Gurusamy, K., Ourselin, S., Davidson, B., Hawkes, D., Clarkson, M.J.: In vivo estimation of target registration errors during augmented reality laparoscopic surgery. Int. J. Comput. Assist. Radiol. Surg. 13(6), 865–874 (2018)CrossRefGoogle Scholar
  22. 22.
    Unity Technologies ApS: Unity3D (2017). Accessed 30 Aug 2018
  23. 23.
    Vávra, P., Roman, J., Zonča, P., Ihnát, P., Němec, M., Kumar, J., Habib, N., El-Gendi, A.: Recent development of augmented reality in surgery: a review. J. Healthcare Eng. 2017, Article ID 4574172, 9 (2017)Google Scholar
  24. 24.
    Yoon, J.W., Chen, R.E., Kim, E.J., Akinduro, O.O., Kerezoudis, P., Han, P.K., Si, P., Freeman, W.D., Diaz, R.J., Komotar, R.J., Pirris, S.M., Brown, B.L., Bydon, M., Wang, M.Y., Wharen, R.E., Quinones-Hinojosa, A.: Augmented reality for the surgeon: systematic review. Int. J. Med. Robot. Comput. Assist. Surg. 14(4), e1914 (2018)CrossRefGoogle Scholar
  25. 25.
    Zhang, J., Zhong, Y., Smith, J., Gu, C.: Energy propagation modeling of nonlinear soft tissue deformation for surgical simulation. Simulation 94(1), 3–10 (2018)CrossRefGoogle Scholar
  26. 26.
    Zheng, Y., Kuang, Y., Sugimoto, S., Åström, K., Okutomi, M.: Revisiting the pnp problem: a fast, general and optimal solution. In: 2013 IEEE International Conference on Computer Vision, pp. 2344–2351, December 2013Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Daniele Amparore
    • 1
  • Enrico Checcucci
    • 1
  • Marco Gribaudo
    • 2
  • Pietro Piazzolla
    • 3
    Email author
  • Francesco Porpiglia
    • 1
  • Enrico Vezzetti
    • 3
  1. 1.Division of Urology, Department of Oncology, School of MedicineUniversity of Turin-San Luigi Gonzaga HospitalTurin, OrbassanoItaly
  2. 2.Dipartimento di Elettronica, Informatica e BioingegneriaPolitecnico di MilanoMilanItaly
  3. 3.Department of Management and Production EngineeringPolitecnico di TorinoTurinItaly

Personalised recommendations