Skip to main content
SpringerLink
Log in

High-precision Calibration of Camera and IMU on Manipulator for Bio-inspired Robotic System

  • Research Article
  • Published:
Journal of Bionic Engineering Aims and scope Submit manuscript

Abstract

Inspired by box jellyfish that has distributed and complementary perceptive system, we seek to equip manipulator with a camera and an Inertial Measurement Unit (IMU) to perceive ego motion and surrounding unstructured environment. Before robot perception, a reliable and high-precision calibration between camera, IMU and manipulator is a critical prerequisite. This paper introduces a novel calibration system. First, we seek to correlate the spatial relationship between the sensing units and manipulator in a joint framework. Second, the manipulator moving trajectory is elaborately designed in a spiral pattern that enables full excitations on yaw–pitch–roll rotations and x–y–z translations in a repeatable and consistent manner. The calibration has been evaluated on our collected visual inertial-manipulator dataset. The systematic comparisons and analysis indicate the consistency, precision and effectiveness of our proposed calibration method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. http://aubo-robotics.cn/Content/382372.html.

  2. https://www.ximea.com/.

  3. https://sourceforge.net/p/vectorduino/home/Home/.

References

  1. Sinatra, N. R., Teeple, C. B., Vogt, D. M., Parker, K. K., Gruber, D. F., & Wood, R. J. (2019). Ultragentle manipulation of delicate structures using a soft robotic gripper. Science Robotics, 4, eaax5425.

    Article  Google Scholar 

  2. Ren, Z. Y., Hu, W. Q., Dong, X. G., & Sitti, M. (2019). Multi-functional soft-bodied jellyfish-like swimming. Nature Communication, 10, 1–12.

    Article  Google Scholar 

  3. Kawano, S., Miyagawa, S., & Shirai, A. (2016). Systematical experiment for optimal design of vibrating flow pump with jelly-fish valve. Journal of Bionic Engineering, 13, 166–179.

    Article  Google Scholar 

  4. Gershwin, L. (2017). Spineless: The science of jellyfish and the art of growing a backbone. Nature, 551, 32–32.

    Article  Google Scholar 

  5. Yuan, Y., Ning, H. L., & Lu, X. Q. (2019). Bio-inspired representation learning for visual attention prediction. IEEE Transactions on Cybernetics, 51, 3562–3575.

    Article  Google Scholar 

  6. Li, D. R., Liu, Y., & Yuan, X. X. (2013). Image-based self-position and orientation method for moving platform. Science China-Information Sciences, 56, 042301.

    MathSciNet  MATH  Google Scholar 

  7. Sun, T. R., Cheng, L., Hou, Z. G., & Tan, M. (2021). Novel sliding-mode disturbance observer-based tracking control with applications to robot manipulators. Science China-Information Sciences, 64, 172205.

    Article  MathSciNet  Google Scholar 

  8. Xing, H. M., Shi, L. W., Tang, K., Guo, S. X., Hou, X. H., Liu, Y., Liu, H. K., & Hu, Y. (2019). Robust rgb-d camera and imu fusion-based cooperative and relative close-range localization for multiple turtle inspired amphibious spherical robots. Journal of Bionic Engineering, 16, 442–454.

    Article  Google Scholar 

  9. Xiao, J. H., Xiong, D., Yu, Q. H., Huang, K. H., Lu, H. M., & Zeng, Z. W. (2020). A real-time sliding-window-based visual-inertial odometry for mavs. IEEE Transactions on Industrial Informatics, 16, 4049–4058.

    Article  Google Scholar 

  10. Luo, B., Chen, H. Y., Quan, F. Y., Zhang, S. W., & Liu, Y. H. (2020). Natural feature-based visual servoing for grasping target with an aerial manipulator. Journal of Bionic Engineering, 17, 215–228.

    Article  Google Scholar 

  11. Zou, D. P., Wu, Y. X., Pei, L., Ling, H. B., & Yu, W. X. (2019). Structvio: Visual-inertial odometry with structural regularity of man-made environments. IEEE Transactions on Robotics, 35, 999–1013.

    Article  Google Scholar 

  12. Sa, I., Kamel, M., Burri, M., Bloesch, M., Khanna, R., Popovic, M., Nieto, J., & Siegwart, R. (2018). Build your own visual-inertial drone: A cost-effective and open-source autonomous drone. IEEE Robotics and Automation Magazine, 25, 89–103.

    Article  Google Scholar 

  13. Liu, Y. Z., & Meng, Z. Y. (2020). Online temporal calibration based on modified projection model for visual-inertial odometry. IEEE Transactions on Instrumentation and Measurement, 69, 5197–5207.

    Article  Google Scholar 

  14. Yang, L., Dryanovski, I., Valenti, R. G., Wolberg, G., & Xiao, J. Z. (2020). Rgb-d camera calibration and trajectory estimation for indoor mapping. Autonomous Robots, 44, 1485–1503.

    Article  Google Scholar 

  15. Skogh, C., Garm, A., Nilsson, D. E., & Ekstrm, P. (2006). Bilaterally symmetrical rhopalial nervous system of the box jellyfish tripedalia cystophora. Mrophology, 267, 1391–1405.

    Article  Google Scholar 

  16. Garm, A., Oskarsson, M., & Nilsson, D. E. (2011). Box jellyfish use terrestrial visual cues for navigation. Current Biology, 21, 798–803.

    Article  Google Scholar 

  17. Kim, D., Kang, B. B., Kim, K. B. K., Chol, H., Ha, J., Cho, K. J., & Jo, S. (2019). Eyes are faster than hands: A soft wearable robot learns user intention from the egocentric view. Science Robotics, 4, eaav2949.

    Article  Google Scholar 

  18. Yagi, T., Mangalam, K., Yonetani, R., & Sato, Y. (2018). Future person localization in first-person videos. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, USA, pp. 7593–7602.

  19. Eckenhoff, K., Yang, Y., Geneva, P., & Huang, G. Q. (2019). Tightly-coupled visual-inertial localization and 3-d rigid-body target tracking. IEEE Robotics and Automation Letters, 4, 1541–1548.

    Article  Google Scholar 

  20. Jin, L. Q., Zhang, H., & Ye, C. (2020). Camera intrinsic parameters estimation by visual inertial odometry for a mobile phone with application to assisted navigation. IEEE/ASME Transactions on Mechatronics, 25, 1803–1811.

    Article  Google Scholar 

  21. Santoso, F., Garratt, M. A., & Anavatti, S. G. (2017). Visual-inertial navigation systems for aerial robotics: Sensor fusion and technology. IEEE Transactions on Automation Science and Engineering, 14, 260–275.

    Article  Google Scholar 

  22. Kim, D., Shin, S., & Kweon, I. S. (2018). On-line initialization and extrinsic calibration of an inertial navigation system with a relative preintegration method on manifold. IEEE Transactions on Automation Science and Engineering, 15, 1272–1285.

    Article  Google Scholar 

  23. Kelly, J., & Sukhatme, G. S. (2011). Visual-inertial sensor fusion: Localization, mapping and sensor-to-sensor self-calibration. The International Journal of Robotics Research, 30, 56–79.

    Article  Google Scholar 

  24. Wang, J. & Olson, E. (2016). Apriltag 2: Efficient and robust fiducial detection. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea, pp. 4193–4198.

  25. Lobo, J., & Dias, J. (2007). Relative pose calibration between visual and inertial sensors. The International Journal of Robotics Research, 26, 561–575.

    Article  Google Scholar 

  26. Forster, C., Carlone, L., Dellaert, F., & Scaramuzza, D. (2017). On-manifold preintegration for real-time visual-inertial odometry. IEEE Transactions on Robotics, 33, 1–21.

    Article  Google Scholar 

  27. Tian, F. P., Feng, W., Zhang, Q., Wang, X. W., Sun, J. Z., Loia, V., & Liu, Z. Q. (2019). Active camera relocalization from a single reference image without hand-eye calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41, 2791–2806.

    Article  Google Scholar 

  28. Choi, Y., Kim, N., Hwang, S., Park, K., Yoon, J. S., An, K., & Kweon, I. S. (2018). Kaist multi-spectral day/night data set for autonomous and assisted driving. IEEE Transactions on Intelligent Transportation Systems, 19, 934–948.

    Article  Google Scholar 

  29. Gao, P. Y., Li, K., Song, T. X., & Liu, Z. J. (2018). An accelerometers-size-effect self-calibration method for tri-axis rotational inertial navigation system. IEEE Transactions Industrial Electronics, 65, 1655–1664.

    Article  Google Scholar 

  30. Xu, W. F., Meng, D. S., Liu, H. D., Wang, X. Q., & Liang, B. (2019). Singularity-free trajectory planning of free-floating multi-arm space robots for keeping the base inertially stabilized. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 49, 2464–2477.

    Article  Google Scholar 

  31. Furgale, P., Rehder, J., & Siegwart, R. (2013) Unified temporal and spatial calibration for multi-sensor systems. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Tokyo, Japan, pp. 1280–1286.

  32. He, H. S., Li, Y., & Tan, J. D. (2016). Rotational coordinate transformation for visual-inertial sensor fusion. International Conference on Social Robotics (ICSR), Qingdao, China, pp. 430–440.

  33. Leutenegger, S., Lynen, S., Bosse, M., Siegwart, R., & Furgale, P. (2015). Keyframe-based visual-inertial odometry using nonlinear optimization. The International Journal of Robotics Research, 34, 314–334.

    Article  Google Scholar 

  34. Li, Y. G., Hou, C., Luo, Y., Zhao, Y. W., & Zhao, X. G. (2021). A arameters identification method for flexible joints based on resonance and anti-resonance frequency characteristics. Robot, 43, 279–288.

    Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (61903357, 61902299, 62022088), the International Partnership Program of Chinese Academy of Sciences (173321KYSB20200002), Liaoning Provincial Natural Science Foundation of China (2020-MS-032, 2021JH6/ 10500114, 2020JH2/10500002), Guangzhou Science and Technology Planning Project (202102021300) and China Postdoctoral Science Foundation (2019TQ0239, 2019M663636).

Author information

Authors and Affiliations

  1. State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, 110016, China

    Yinlong Zhang, Wei Liang, Sichao Zhang & Xudong Yuan

  2. Key Laboratory of Networked Control Systems, Chinese Academy of Sciences, Shenyang, 110016, China

    Yinlong Zhang, Wei Liang, Sichao Zhang & Xudong Yuan

  3. Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang, 110169, China

    Yinlong Zhang, Wei Liang, Sichao Zhang & Xudong Yuan

  4. The School of Computer Science and Technology, Xidian University, Xi’an, 710071, China

    Xiaofang Xia

  5. Department of Mechanical, Aerospace and Biomedical Engineering, University of Tennessee, Konxville, 37996, USA

    Jindong Tan

  6. Department of Automation Technology, ABB Corporate Research Sweden, 72178, Västerås, Sweden

    Zhibo Pang

Authors
  1. Yinlong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sichao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xudong Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaofang Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jindong Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Zhibo Pang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Liang.

Ethics declarations

Conflict of interest

The authors declare that there are no conflicts of interest to this work.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Y., Liang, W., Zhang, S. et al. High-precision Calibration of Camera and IMU on Manipulator for Bio-inspired Robotic System. J Bionic Eng 19, 299–313 (2022). https://doi.org/10.1007/s42235-022-00163-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42235-022-00163-7

Keywords

Search

Navigation