Journal of Micro-Bio Robotics

, Volume 11, Issue 1–4, pp 35–45 | Cite as

Vision-based haptic feedback for capsule endoscopy navigation: a proof of concept

  • Marco MuraEmail author
  • Yasmeen Abu-Kheil
  • Gastone Ciuti
  • Marco Visentini-Scarzanella
  • Arianna Menciassi
  • Paolo Dario
  • Jorge Dias
  • Lakmal Seneviratne
Research Paper


In this paper, a vision-based haptic feedback system has been proposed with the aim to assist the movement of an endoscopic device during capsule endoscopy (CE) procedures. We present a general system architecture consisting of three modules responsible for vision, haptic guidance and control of movements. The vision module generates 3D local maps as well as local navigation trajectory for endoluminal navigation. The haptic guidance module consists of a haptic device that allows the user to control the movement of the capsule along the generated path. The haptics module also helps the operator by transforming the 3D maps and the relative paths into a guiding virtual force. Measuring the current relative distance between the user input and the maps boundaries, the haptic guidance module will check if the user is moving away or toward the colonic walls and will generate a feedback force with the aim to assist the operator during the navigation procedure. The user will also sense an attractive virtual feedback force toward the generated path that will help the user in the navigation. Finally, the movement control module is the interface between the haptics module and the chosen manipulator. The final goal is to develop a complete active CE robotic platform with haptic feedback in order to enhance safety, to reduce cost (using the same system as a training simulator as well as real endoscopic platform) and to help the operator during the navigation by combining all 3D local maps into a full 3D reconstructed colon.


Colonoscopy Haptic guidance Robotic endoscopic capsule 3D reconstruction Vision-based haptic feedback 



The authors would like to thank Reem Ashour (Khalifa University Robotic Institute, Abu Dhabi, UAE) for her suggestions and support in the development of the Gazebo environment and in the development of the haptic force field generation.


  1. 1.
    World health organization. [Online]. Available:, [May 20, 2016]
  2. 2.
    International agency for research on cancer - estimated incidence, mortality and prevalence worldwide in 2012. [Online]. Available:, [May 20, 2016]
  3. 3.
    Cancer research uk. [Online]. Available:, [May 20, 2016]
  4. 4.
    Sliker LJ, Ciuti G (2014) “Flexible and capsule endoscopy for screening, diagnosis and treatment,”. Expert Rev Med Dev 11(6):649–666CrossRefGoogle Scholar
  5. 5.
    Fan Y, Meng M-H, Li B (2010) 3D reconstruction of wireless capsule endoscopy images. In: Annual international conference of the IEEE engineering in medicine and biology society (EMBC), pp 5149–5152Google Scholar
  6. 6.
    Sliker L, Ciuti G, Rentschler M, Menciassi A (2015) Magnetically driven medical devices: a review. Expert Rev Med Dev 12(6):737–752CrossRefGoogle Scholar
  7. 7.
    Sliker LJ, Kern MD, Schoen JA, Rentschler ME (2012) Surgical evaluation of a novel tethered robotic capsule endoscope using micro-patterned treads. Surg Endos 26(10):2862–2869CrossRefGoogle Scholar
  8. 8.
    Valdastri P, Webster RJ III, Quaglia C, Quirini M, Menciassi A, Dario P (2009) A new mechanism for mesoscale legged locomotion in compliant tubular environments. IEEE Trans Robot 25(5):1047–1057CrossRefGoogle Scholar
  9. 9.
    Carpi F, Pappone C (2009) Magnetic maneuvering of endoscopic capsules by means of a robotic navigation system. IEEE Trans Biomed Eng 56(5):1482–1490CrossRefGoogle Scholar
  10. 10.
    Ciuti G, Valdastri P, Menciassi A, Dario P (2010) Robotic magnetic steering and locomotion of capsule endoscope for diagnostic and surgical endoluminal procedures. Robotica 28(02):199– 207CrossRefGoogle Scholar
  11. 11.
    Lucarini G, Mura M, Ciuti G, Rizzo R, Menciassi A (2015) Electromagnetic control system for capsule navigation: Novel concept for magnetic capsule maneuvering and preliminary study. J Med Biol Eng 35(4):428–436CrossRefGoogle Scholar
  12. 12.
    Lucarini G, Ciuti G, Mura M, Rizzo R, Menciassi A (2015) A new concept for magnetic capsule colonoscopy based on an electromagnetic system. Int J Adv Robot Syst 12:25Google Scholar
  13. 13.
    Ciuti G, Donlin R, Valdastri P, Arezzo A, Menciassi A, Morino M, Dario P et al (2010) Robotic versus manual control in magnetic steering of an endoscopic capsule. Endoscopy 42(2):148CrossRefGoogle Scholar
  14. 14.
    Okamura AM (2004) Methods for haptic feedback in teleoperated robot-assisted surgery. Ind Robot Int J 31 (6):499–508CrossRefGoogle Scholar
  15. 15.
    Reilink R, Stramigioli S, Kappers AM, Misra S (2011) Evaluation of flexible endoscope steering using haptic guidance. Int J Med Robot Comput Assist Surg 7(2):178–186CrossRefGoogle Scholar
  16. 16.
    Ghanbari A, Horan B, Nahavandi S, Chen X, Wang W (2014) Haptic microrobotic cell injection system. IEEE Syst J 8(2):371–383CrossRefGoogle Scholar
  17. 17.
    Pacchierotti C, Magdanz V, Medina-Sánchez M, Schmidt O G, Prattichizzo D, Misra S (2015) Intuitive control of self-propelled microjets with haptic feedback. J Micro-Bio Robot 10(1–4):37–53CrossRefGoogle Scholar
  18. 18.
    Mehrtash M, Khamesee MB, Tarao S, Tsuda N, Chang J-Y (2012) Human-assisted virtual reality for a magnetic-haptic micromanipulation platform. Microsyst Technol 18(9–10):1407–1415CrossRefGoogle Scholar
  19. 19.
    Maier-Hein L, Groch A, Bartoli A, Bodenstedt S, Boissonnat G, Chang P-L, Clancy N, Elson DS, Haase S, Heim E et al (2014) Comparative validation of single-shot optical techniques for laparoscopic 3-d surface reconstruction. IEEE Trans Med Imag 33(10):1913–1930CrossRefGoogle Scholar
  20. 20.
    Maier-Hein L, Mountney P, Bartoli A, Elhawary H, Elson D, Groch A, Kolb A, Rodrigues M, Sorger J, Speidel S et al (2013) Optical techniques for 3d surface reconstruction in computer-assisted laparoscopic surgery. Med Image Anal 17(8):974–996CrossRefGoogle Scholar
  21. 21.
    Ciuti G, Visentini-Scarzanella M, Dore A, Menciassi A, Dario P, Yang G-Z (2012) Intra-operative monocular 3D reconstruction for image-guided navigation in active locomotion capsule endoscopy. In: 4th IEEE RAS & EMBS international conference on biomedical robotics and biomechatronics (BioRob), pp 768–774Google Scholar
  22. 22.
    Siciliano B, Khatib O (2008) Springer handbook of robotics. Springer, BerlinCrossRefzbMATHGoogle Scholar
  23. 23.
    Linner T, Shrikathiresan A, Vetrenko M, Ellmann B Modeling and operating robotic environent using gazebo/ros. In: Proceedings of the 28th international symposium on automation and robotics in construction (ISARC2011), pp 957–962Google Scholar
  24. 24.
    Bouguet JY (2004) Camera calibration toolbox for matlabGoogle Scholar
  25. 25.
    Engel J, Stückler J, Cremers D (2015) Large-scale direct slam with stereo cameras. In: Proceedings of the IEEE international conference on intelligent robots and systems (IROS)Google Scholar
  26. 26.
    Mur-Artal R, Montiel J, Tardos J (2015) Orb-slam: a versatile and accurate monocular slam system. IEEE Trans Robot 31(5):1147–1163CrossRefGoogle Scholar
  27. 27.
    Visentini-Scarzanella M, Stoyanov D, Yang G-Z (2012) Metric depth recovery from monocular images using shape-from-shading and specularities. In: 2012 19th IEEE international conference on image processing (ICIP). IEEE, pp 25–28Google Scholar
  28. 28.
    Hast A, Nysjö J, Marchetti A (2013) Optimal ransac-towards a repeatable algorithm for finding the optimal setGoogle Scholar
  29. 29.
    Quigley M, Conley K, Gerkey B, Faust J, Foote T, Leibs J, Wheeler R, Ng AY (2009) Ros: an open-source robot operating system. ICRA Workshop Open Source Softw 3(3.2):5Google Scholar
  30. 30.
    Chitta S, Sucan I, Cousins S (2012) Moveit![ros topics]. IEEE Robot Autom Mag 1(19):18–19CrossRefGoogle Scholar
  31. 31.
    Abu-Kheil Y, Ciuti G, Mura M, Dias J, Dario P, Seneviratne L (2015) Vision and inertial-based image mapping for capsule endoscopy. In: 2015 International conference on information and communication technology research (ICTRC). IEEE, pp 84–87Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Marco Mura
    • 1
    Email author
  • Yasmeen Abu-Kheil
    • 2
  • Gastone Ciuti
    • 1
  • Marco Visentini-Scarzanella
    • 3
  • Arianna Menciassi
    • 1
  • Paolo Dario
    • 1
  • Jorge Dias
    • 2
  • Lakmal Seneviratne
    • 2
  1. 1.The BioRobotics InstituteScuola Superiore Sant’Anna, PontederaPisaItaly
  2. 2.Khalifa University Robotics InstituteKhalifa University of Science, Technology and ResearchAbu DhabiUAE
  3. 3.Computer Vision and Graphics LaboratoryKagoshima UniversityKagoshimaJapan

Personalised recommendations