Fusion of Inertial Measurements and Vision Feedback for Microsurgery

  • Yan Naing Aye
  • Su Zhao
  • Cheng Yap Shee
  • Wei Tech Ang
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 194)


A microsurgery system that achieves real-time enhanced micrometer scale positioning accuracy by fusing visual information from a high speed monovision camera mounted on an optical surgical microscope and acceleration measurements from an intelligent handheld instrument, ITrem2, is presented. The high speed camera captures images of the tool tip of ITrem2 to track its position in real-time. The focus value of the tool tip in the acquired image is used to locate the tool tip along the principal axis of the objective lens of the microscope and edge based geometric template matching gives the position in pixel coordinates. ITrem2 utilizes four dual-axis miniature digital MEMS accelerometers to sense and update the motion information. The system has a first in, first out (FIFO) queue to track the recent history of the slow non-drifting position estimation from the vision system and acceleration readings from the inertial sensors together with their respective time stamps. In the proposed method, real-time visual servoing of micrometer scale motion is achieved by taking into account the dynamic behavior of the vision feedback and incorporating synchronized fusion of these complementary sensors.


sensor fusion visual servoing microsurgery 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Rehbinder, H., Ghosh, B.K.: Multi-rate fusion of visual and inertial data. In: International Conference on Multisensor Fusion and Integration for Intelligent Systems, MFI 2001, pp. 97–102 (2001)Google Scholar
  2. 2.
    Armesto, L., Chroust, S., Vincze, M., Tornero, J.: Multi-rate fusion with vision and inertial sensors. In: Robotics and Automation, Proceedings of ICRA 2004, April 26 - May 1, vol. 1, pp. 193–199 (2004)Google Scholar
  3. 3.
    Hol, J.D.: Robust real-time tracking by fusing measurements from inertial and vision sensors. Journal of Real-Time Image Processing 2, 149–160 (2007)CrossRefGoogle Scholar
  4. 4.
    Huster, A., Frew, E.W., Rock, S.M.: Relative position estimation for AUVs by fusing bearing and inertial rate sensor measurements. In: OCEANS 2002 MTS/IEEE, October 29-31, vol. 3, pp. 1863–1870 (2002)Google Scholar
  5. 5.
    Parnian, N., Golnaraghi, M.F.: Integration of vision and inertial sensors for industrial tools tracking. Sensor Review 27(2), 132–141 (2007)CrossRefGoogle Scholar
  6. 6.
    Tan, U.-X., Veluvolu, K., Latt, W.T., Shee, C.Y., Riviere, C., Ang, W.T.: Estimating displacement of periodic motion with inertial sensors. IEEE Sensors Journal 8, 1385–1388 (2008)CrossRefGoogle Scholar
  7. 7.
    Veluvolu, K.C., Ang, W.T.: Estimation of physiological tremor from accelerometers for real-time applications. Sensors 11(3), 3020–3036 (2011)CrossRefGoogle Scholar
  8. 8.
    Latt, W.T., Veluvolu, K.C., Ang, W.T.: Drift-free position estimation of periodic or quasi-periodic motion using inertial sensors. Sensors 11(6), 5931–5951 (2011)CrossRefGoogle Scholar
  9. 9.
    Karras, G.C., Loizou, S.G., Kyriakopoulos, K.J.: A visual-servoing scheme for semi-autonomous operation of an underwater robotic vehicle using an IMU and a Laser Vision System. In: Robotics and Automation, ICRA 2010, May 3-7, pp. 5262–5267 (2010)Google Scholar
  10. 10.
    Zhou, Y., Nelson, B.J., Vikramaditya, B.: Fusing force and vision feedback for micromanipulation. In: Proceedings of 1998 IEEE International Conference on Robotics and Automation, May 16-20, vol. 2, pp. 1220–1225 (1998)Google Scholar
  11. 11.
    Yu Sun, S., Duthaler, S., Nelson, B.J.: Autofocusing Algorithm Selection in Computer Microscopy. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, August 2-6, pp. 70–76 (2005)Google Scholar
  12. 12.
    Wu, Q., Merchant, F.A., Castleman, K.R.: Microscope Image Processing. Academic Press (2008)Google Scholar
  13. 13.
    Ang, W.T., Riviere, C.N., Khosla, P.K.: Nonlinear regression model of a low-g MEMS accelerometer. IEEE Sensors J. 7(1&2), 81–88 (2007)CrossRefGoogle Scholar
  14. 14.
    Latt, W.T., Tan, U.-X., Shee, C.Y., Riviere, C.N., Ang, W.T.: Compact Sensing Design of a Hand-held Active Tremor Compensation Instrument. IEEE Sensors Journal 9, 1864–1871 (2009)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Yan Naing Aye
    • 1
  • Su Zhao
    • 1
  • Cheng Yap Shee
    • 1
  • Wei Tech Ang
    • 1
  1. 1.School of Mechanical and Aerospace EngineeringNanyang Technological UniversityNanyangSingapore

Personalised recommendations