Virtual Reality-based Control of Robotic Endoscope in Laparoscopic Surgery

  • Yeeun Jo
  • Yoon Jae Kim
  • Minwoo Cho
  • Chiwon Lee
  • Myungjoon Kim
  • Hye-Min Moon
  • Sungwan KimEmail author


Robotic laparoscopic surgery has provided various benefits, but during the surgery, the surgeons are experiencing uncomfortable positioning issue which leads to neck pain and back pain. So, for an enhanced & efficient ergonomic robot-assisted surgery, a novel virtual reality (VR)-based endoscope control system (ECS) is proposed in this research. The overall system in this study is composed of a da Vinci research kit (dVRK), 4-degree-of-freedom ECS, three-dimensional endoscope, and VR headset with a built-in attitude and heading reference system (AHRS) module. Also, the proposed VR headset could replace a stereo viewer in the dVRK which could reduce the size of surgical robot system. Furthermore, the three dimensional endoscope could be controlled by a built-in AHRS module in the VR headset. The proposed system has been verified with four novice volunteers. They showed rapid learning through the peg-transfer task. Additionally, a collision avoidance strategy for VR-based ECS control was developed and verified by performing computer-based simulations. The ergonomic VR-based ECS proposed in this research could greatly reduce surgeon’s pains in both neck and back which potentially reduce surgeon’s workload during the surgery.


Collision avoidance dVRK ergonomic system robot-assisted surgery virtual reality 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



  1. [1]
    G. H. Ballantyne, “Robotic surgery, telerobotic surgery, telepresence, and telementoring,” Surgical Endoscopy And Other Interventional Techniques, vol. 16, no. 10, pp. 1389–1402, October 2002.CrossRefGoogle Scholar
  2. [2]
    A. T. Ng and P. C. Tam, “Current status of robot-assisted surgery,” Hong Kong Medical Journal, vol. 20, no. 3, pp. 241–250, May 2014.Google Scholar
  3. [3]
    T. E. Ahlering, D. Sharecky, D. Lee, and R. V. Clayman, “Successful transfer of open surgical skills to a laparoscopic environment using a robotic interface: initial experience with laparoscopic radical prostatectomy,” The Journal of Urology, vol. 170, no. 5, pp. 1738–1741, November 2003.CrossRefGoogle Scholar
  4. [4]
    V. Ficarra, G. Novara, W. Artibani, A. Cestari, A. Galfano, M. Graefen, G. Guazzoni, B. Guillonneau, M. Menon, F. Montorsi, V. Patel, J. Rassweiler, and H. Van Poppel, “Retropubic, Laparoscopic, and Robot-Assisted Radical Prostatectomy: A system review and cumulative analysis of comparative studies,” European Urology, vol. 55, no. 5, pp. 1037–1063, May 2009.CrossRefGoogle Scholar
  5. [5]
    J. P. Ruurda, T. J. van Vroonhoven, and I. A. Broeders, “Robot-assisted surgical systems: a new era in laparoscopic surgery,” Annals of the Royal College of Surgeons of England, vol. 84, no. 4, pp. 223–226, July 2002.CrossRefGoogle Scholar
  6. [6]
    J. Finkelstein, E. Eckersberger, H. Sadri, S. S. Taneja, H. Lepor, and B. Djavan, “Open versus laparoscopic versus robot-assisted laparoscopic prostatectomy: The European and US experience,” Reviews in Urology, vol. 12, no. 1, pp. 35–43, 2010.Google Scholar
  7. [7]
    O. A. J. Van der Meijden and M. P. Schijven, “The value of haptic feedback in conventional and robot-assisted minimal invasive surgery and virtual reality training: a current review,” Surgical Endoscopy, vol. 23, no. 6, pp. 1180–1190, June 2009.CrossRefGoogle Scholar
  8. [8]
    G. Turchetti, F. Pierotti, and A. Cuschieri, “Economic evaluation of da Vinci-assisted robotic surgery: a systematic review,” Surgical Endoscopy, vol. 26, no. 3, pp. 598–606, March 2012.CrossRefGoogle Scholar
  9. [9]
    C. W. Kim, C. H. Kim, and S. H. Baik, “Outcomes of robotic-assisted colorectal surgery compared with laparoscopic and open surgery: a systematic review,” Journal of Gastrointestinal Surgery, vol. 18, no. 4, pp. 816–830, April 2014.CrossRefGoogle Scholar
  10. [10]
    A. Ahmad, ZF. Ahmad, JD. Carleton, and A. Agarwala, “Robotic surgery: current perceptions and the clinical evidence,” Surgical Endoscopy, vol. 31, no. 1, pp. 255–263, January 2017.CrossRefGoogle Scholar
  11. [11]
    J. B. Gomez, A. Ceballos, F. Prieto, and T. Redarce, “Mouth gesture and voice command based robot command interface,” Proc. of the Conf. 2009 IEEE International Conference on Robotics and Automation (ICRA), July 2009.Google Scholar
  12. [12]
    N. Hong, M. Kim, C. Lee, and S. Kim, “Head-mounted interface for intuitive vision control and continuous surgical operation in a surgical robot system,” Medical & Biological Engineering & Computing, vol. 57, no. 3, pp. 601–614, March 2019.CrossRefGoogle Scholar
  13. [13]
    M. Kim, C. Lee, W. Park, Y. S. Suh, H. K. Yang, H. J. Kim, and S. Kim, “A development of assistant surgical robot system based on surgical operation-by-wire and hands-on-throttle-and-stick,” Biomedical Engineering Online, vol. 15, no. 58, May 2016.Google Scholar
  14. [14]
    M. Kim, C. Lee, N. Hong, Y. J. Kim, and S. Kim, “Development of stereo endoscope system with its innovative master interface for continuous surgical operation,” Biomedical Engineering Online, vol. 16, no. 81, pp. 1–16, June 2017.Google Scholar
  15. [15]
    Y. Cao, S. Miura, Y. Kobayashi, K. Kawamura, S. Sugano, and M. G. Fujie, “Pupil variation applied to the eye tracking control of an endoscopic manipulator,” IEEE Robotics and Automation Letter, vol. 1, no. 1, pp. 531–538, January 2016.CrossRefGoogle Scholar
  16. [16]
    T. Kawai, M. Fukunish, A. Nishikawa, Y. Nishizawa, and T. Nakamura, “Hands-free interface for surgical procedures based on foot movement patterns,” Proc. of the 36th Conf. Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 345–348, November 2014.Google Scholar
  17. [17]
    M. M. Lux, M. Marshall, E. Erturk, and J. V. Joseph, “Ergonomic Evaluation and Guidelines for Use of the DaVinci Robot System,” Journal of Endourology, vol. 24, no. 3, March 2010.CrossRefGoogle Scholar
  18. [18]
    Oculus VR, “Initialization and Sensor Enumeration,”, Accessed May 29, 2018.
  19. [19]
    Y. Jo, Y. J. Kim, H.-M. Moon, and S. Kim, “Development of Virtual Reality-Vision System in Robot-Assisted Laparoscopic Surgery,” Proc. of the 18th Conf. International Conference on Control, Automation and System, pp. 1708–1712, 2018.Google Scholar
  20. [20]
    K. Zinchenko, O. Komarov, and K. T. Song, “Virtual reality control of a robotic camera holder for minimally invasive surgery,” Proc. of the 11th Conf. Asian Control Conference (ICROS), pp. 970–975, February 2017.Google Scholar
  21. [21]
    Fundamentals of Laparoscopic Surgery, “FLS manual skills written instructions and performance guideline,”, Accessed January 2019.
  22. [22]
    Y. J. Kim, H. S. Nam, W. H. Lee, H. G. Seo, J.-H. Leigh, B.-M. Oh, M. S. Bang, S. Kim, “Vision-aided brain-machine interface training system for robotic arm control and clinical application on two patients with cervical spinal cord injury,” BioMedical Engineering Online, vol. 18, no. 14, February 2019.Google Scholar
  23. [23]
    Y. Qiu, Z. Yan, Y. Miao, and Z. Du, “Real-time collision avoidance algorithm for surgical robot based on OBB intersection test,” Proc. of the 19th Conf. International Conference on Intelligent Robotics and Applications, pp. 195–205, August 2017.Google Scholar

Copyright information

© ICROS, KIEE and Springer 2019

Authors and Affiliations

  • Yeeun Jo
    • 1
  • Yoon Jae Kim
    • 2
  • Minwoo Cho
    • 1
  • Chiwon Lee
    • 3
  • Myungjoon Kim
    • 3
  • Hye-Min Moon
    • 1
  • Sungwan Kim
    • 4
    • 5
    Email author
  1. 1.Program for BioengineeringSeoul National UniversitySeoulKorea
  2. 2.Institute of Medical and Biological EngineeringSeoul National UniversitySeoulKorea
  3. 3.Korea Electrotechnology Research InstituteAnsanKorea
  4. 4.Institute of Medical and Biological EngineeringSeoul National UniversitySeoulKorea
  5. 5.Department of Biomedical EngineeringSeoul National University College of MedicineSeoulKorea

Personalised recommendations