Skip to main content
SpringerLink
Log in

Three Dimensional Microrobot Tracking Using Learning-based System

  • Published:
International Journal of Control, Automation and Systems Aims and scope Submit manuscript

Abstract

As a promising method for robotic catheter therapeutics, a controllable wireless microrobot system with a less invasive intraoperative procedure has been presented for thrombosis surgery. The controllable wireless microrobot utilizes an electromagnetic actuator system for microrobot actuation and biplane X-ray system for microrobot imaging. We propose the new 3D tracking method of microrobot for this system. As a learning-based system, cascade classifier was adapted for real-time microrobot tracking. Combination of cascade classifier and contour-based system made accurate microrobot detection method on 2D projected X-ray images. Kalman filter interpolated lost frames, and triangulation method reconstructed 3D pose (position and orientation) from 4 endpoints of microrobot pairs. Also, the semi-automatic calibration method for bi-plane C-arm devices was proposed by using modified shortest path algorithm. To find the optimal rescaling setting, 3D tracking accuracy was investigated according to the rescaling of original intensity values. The experiment results showed a good performance with tracking errors of 2.37±9.91mm in position and 6.53±13.80° in orientation under 2200 optimal width of rescaling. If tracking evaluation is constrained under frames detected by cascade classifier, 3D tracking errors improved sincerely by 0.28±1.13mm in position and 3.48±2.89° in orientation. The optimal width of the rescaling setting needed bigger value than the mean of intensity values. The proposed tracking technique accomplished a fast frame rate of 34.72 frames/sec under OpenCL implementation of OpenCV. Learning based system is robust to the change of graphics setting since training can adapt rapidly to this change. Also, proposed framework can cooperate with the change of microrobot shape if a contour-based method is optimized. Hence, the method can be used for therapeutic millimeter- or micron-sized manipulator recognition in vascular, as well as implanted objects in the human body.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. K. Kearney, R. S. Hira, R. F. Riley, A. Kalyanasundaram, and W. L. Lombardi, “Update on the management of chronic total occlusions in coronary artery disease,” Current Atherosclerosis Reports, vol. 19, no. 3, pp. 1–10, April 2017.

    Google Scholar 

  2. E. J. Benjamin, M. J. Blaha, and S. E. Chiuve, M. Cushman, S. R. Das, S. D. de Ferranti, and J. Floyd, “Heart disease and stroke statistics-2017 update: a report from the american heart association,” Circulation, vol. 135, no. 10, pp. e146–e603, January 2017.

    Article  Google Scholar 

  3. M. P. Kummer, J. J. Abbott, B. E. Kratochvil, R. Border, A. Sengul, and B. J. Nelson, “OctoMag: an electromagnetic system for 5-DOF wireless micromanipulation,” IEEE Transation on Robotics, vol. 26, no. 6, pp. 1006–1017, December 2010.

    Article  Google Scholar 

  4. F. Ullrich, C. Bergeles, J. Pokki, O. Ergeneman, S. Erni, G. Chatzipirpiridis, S. Pané, C. Framme, and B. J. Nelson “Mobility experiments with microrobots for minimally invasive intraocular surgery,” Investigative Ophthalmology & Visual Science, vol. 54, no. 4, pp. 2853–2863, April 2013.

    Article  Google Scholar 

  5. B. J. Nelson, I. K. Kaliakatsos, and J. J. Abbott, “Microrobots for minimally invasive medicine,” Annual Review of Biomedical Engineering, vol. 12, pp. 55–85, August 2010.

    Article  Google Scholar 

  6. S. Jeon, A. K. Hoshiar, K. Kim, S. Lee, E. Kim, S. Lee, J. Kim, B. J. Nelson, H.-J. Cha, B.-J. Yi, and H. Choi, “A magnetically controlled soft microrobot steering a guidewire in a three-dimensional phantom vascular network,” Soft Robotics, vol. 6, No. 1, online, February 2019.

    Google Scholar 

  7. R. Bashir, C. J. Zack, H. Zhao, A. J. Comerota, and A. A. Bove, “Comparative outcomes of catheter-directed thrombolysis plus anticoagulation vs anticoagulation alone to treat lower-extremity proximal deep vein thrombosis,” JAMA International Medicine, vol. 174, no. 9, pp. 1494–1501, September 2014.

    Article  Google Scholar 

  8. L. Song and M. Takatsuka, “Real-time 3D finger pointing for an augmented desk,” Proceedings of the Sixth Australasian conference on User interface, vol. 40, pp. 99–108, January 2005.

    Google Scholar 

  9. S. S. Rautaray and A. Agrawal, “Vision based hand gesture recognition for human computer interaction: a survey,” Artificial Intelligence Review, vol. 43, no. 1, pp. 1–54, January 2015.

    Article  Google Scholar 

  10. P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 511–518, December 2001.

    Google Scholar 

  11. R. Lienhart and J. Maydt, “An extended set of haar-like features for rapid object detection,” Proc. of International Conference on Image Processing, pp. 900–903, September 2002.

    Chapter  Google Scholar 

  12. R. E. Schapire, “The boosting approach to machine learning: an overview,” Nonlinear Estimation and Classification. Lecture Notes in Statistics, vol. 171, Springer, New York, NY., 2003.

    Chapter  Google Scholar 

  13. Opencv Library. http://www.opencv.org

  14. B. Wang, Z. Qi, S. Chen, Z. Liu, and G. Ma, “Multivehicle detection with identity awareness using cascade adaboost and adaptive Kalman filter for driver assistant system,” PLoS One, vol. 12, no. 3, e0173424, March 2017.

    Google Scholar 

  15. S. Sivaraman and M. M. Trivedi, “Active learning based robust monocular vehicle detection for on-road safety systems,” Proc. of IEEE Intelligent Vehicles Symposium, pp. 399–404, June 2009.

    Google Scholar 

  16. B. M. Kelm, M. Wels, S. K. Zhou, S. Seifert, M. Suehling, Y. Zheng, and D. Comaniciu, “Spine detection in CT and MR using iterated marginal space learning,” Medical Image Analysis, vol. 17, no. 8, pp. 1283–1292, December 2013.

    Article  Google Scholar 

  17. M. Vania, D. Mureja, and D. Lee, “Automatic spine segmentation from CT images using convolutional neural network via redundant generation of class labels,” Journal of Computational Design and Engineering, online, February 2019.

    Google Scholar 

  18. F. Albiol, A. Corbi, and A. Albiol, “Evaluation of modern camera calibration techniques for conventional diagnostic X-ray imaging settings,” Radiological Physics and Technology, vol. 10, no. 1, pp. 68–81, March 2017.

    Article  Google Scholar 

  19. P. B. Nguyen, B. Kang, and D. M. Bappy, E. Choi, S. Park, S. Y. Ko, and J.-O. Park, “Real-time microrobot posture recognition via biplane X-ray imaging system for external electromagnetic actuation,” International Journal of Computer Assisted Radiology and Surgery, vol 13, no. 11, pp 1843–1852, November 2018.

    Article  Google Scholar 

  20. J. P. Bae, Advanced Segmentation Algorithms Using Numerical Model and Graph Theory-application to the Lung Images, Ph.D. Thesis, Seoul National University, February 2016.

    Google Scholar 

  21. D. M. Mount and S. Arya, “ANN: a library for approximate nearest neighbor searching,” Available: http://www.cs.umd.edu/mount/ANN/

  22. X. S. Gao, X. R. Hou, J. Tang, and H. F. Cheng, “Complete solution classification for the perspective-three-point problem,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, no. 8, pp.930–943, August 2003.

    Article  Google Scholar 

  23. Z. Zhang, “A flexible new technique for camera calibration,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp.1330–1334, November 2000.

    Article  Google Scholar 

  24. R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, Cambridge Unversity Press, New York, NY, 2004.

    Book  Google Scholar 

  25. G. Welch and G. Bishop, An Introduction to the Kalman Filter, Technical Report, University of North Carolina at Chapel Hill, NC, USA, 1995.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Center for Medical Robotics, Korea Institute of Science and Technology, 5, Hwarang-ro 14-gil, Seongbuk-gu, Seoul, 02792, Korea

    Jang Pyo Bae, Siyeop Yoon, Malinda Vania & Deukhee Lee

  2. Division of Bio-medical Science & Technology, KIST School, Korea University of Science and Technology, Seoul, 02792, Korea

    Siyeop Yoon, Malinda Vania & Deukhee Lee

Authors
  1. Jang Pyo Bae
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Siyeop Yoon
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Malinda Vania
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Deukhee Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Deukhee Lee.

Additional information

Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Recommended by Guest Editors Doo Yong Lee (KAIST) and Jaesoon Choi (Asan Medical Center). This work was supported by the Ministry of Trade, Industry and Energy of Korea (Grant no. 10052980).

Jang Pyo Bae received his Ph.D. degree in Biomedical Engineering from Seoul National University in 2016. From 2016 to 2017, He was a Ph.D. associate researcher in CISTIB Lab of Sheffield University, UK. He is currently working as a Ph.D. researcher in Korea Institute of Science and Technology. His research interests include robotic surgery planning, image segmentation, volume registration.

Siyeop Yoon received his BA in Biomedical Engineering at Yonsei University, Korea, in 2014. He is currently a Ph.D. candidate in Biomedical Engineering at the University of Science and Technology, Korea. From 2017 to 2018, He was a visiting researcher in Visual intelligence Lab of Iowa University, US. His research interests are image registration, deformable shape modeling, and machine learning.

Malinda Vania received a BE in Informatics Engineering from Ma Chung University, Indonesia, in 2011. She is currently a Ph.D. candidate in Biomedical Engineering at the University of Science and Technology, Korea. Her research interests are medical image processing, image segmentation, image reconstruction and machine learning.

Deukhee Lee received his Ph.D. degree in Mechanical Engineering from the University of Tokyo, Japan in 2008. He is currently working as a principal researcher in Korea Institute of Science.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bae, J.P., Yoon, S., Vania, M. et al. Three Dimensional Microrobot Tracking Using Learning-based System. Int. J. Control Autom. Syst. 18, 21–28 (2020). https://doi.org/10.1007/s12555-019-0241-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12555-019-0241-z

Keywords

Search

Navigation