Skip to main content
SpringerLink
Log in

Visual identification and pose estimation algorithms of nut tightening robot system

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

To realize the automatic tightening of nuts, a nut tightening robot system strategy based on machine vision was designed in this work. The strategy was based on three stages: nut image calibration, nut identification, and nut pose estimation. In the first stage, the template pose image of the nut and the coordinates of the nut center in this nut image were obtained by calibration. In the second stage, a nut identification algorithm based on improved the backbone feature extraction network and area generation network of Faster-RCNN was presented, which improved the efficiency and accuracy of nut identification. In the last stage, a nut pose estimation algorithm based on Fourier and log-polar coordinate transformation was presented to solve the rotation angle, translation, and scale of the nut relative to the template nut image and then obtain the rotation angle of the sleeve and the central coordinate of the nut. An experimental nut tightening robot platform was also set up in this work. The results of 50 tests showed that the proposed detection methods could identify nuts with 100% accuracy, and with the proposed pose estimation methods, the average error of the rotation angle of the nut was 0.057°, and the average error of the center position of the nut in \(x\) and \(y\) directions was ± 0.05 mm and of \(z\) direction was ± 0.5 mm. The experimental results showed that the nut tightening robot scheme and algorithm designed in this work were feasible in nut identification and pose estimation and met the requirements of insertion accuracy in the process of nut tightening.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

Data availability

Not applicable.

Code availability

Code is available when required.

References

  1. Roda-Sanchez L, Garrido-Hidalgo C, García AS et al (2021) Comparison of RGB-D and IMU-based gesture recognition for human-robot interaction in remanufacturing. Int J Adv Manuf Technol. https://doi.org/10.1007/s00170-021-08125-9

    Article  Google Scholar 

  2. Zou BC, Qiu HD, Lu YF (2021) Spherical object recognition based on the number of contour edges extracted by fitting and convex hull processing. Proc Inst Mech Eng, Part D: J Automob Eng 235(5):1225–1235

    Article  Google Scholar 

  3. Cristian V, Silvia K, Benny T (2021) Processing chain for 3d histogram of gradients based real-time object recognition. Int J Adv Rob Syst 18(1):1–13

    Google Scholar 

  4. Peng YQ, Zhao XS, Tao HF et al (2019) Gesture recognition based on deep learning in complex background. Robot 41(4):534–542

    Google Scholar 

  5. He K, Zhang X, Ren S et al (2015) Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell 37(9):1904–1916

    Article  Google Scholar 

  6. Girshick R (2015) Fast R-CNN. Proceedings of the IEEE international conference on computer vision. IEEE, pp 1440–1448.

  7. Ren S, He K, Girshick R et al (2017) Faster R-CNN: towards real-time object detection with region proposal networks [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence 39(06):1137–1149

  8. He K, Ckioxari G, Dollar P, et al. (2020) Mask R-CNN [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence. 42(02):386–397

  9. Wang Z, Xu Y, He Q et al (2020) Grasping pose estimation for SCARA robot based on deep learning of point cloud. Int J Adv Manuf Technol 108:1217–1231. https://doi.org/10.1007/s00170-020-05257-2

    Article  Google Scholar 

  10. Wang Z, Fan J, Jing F et al (2019) A pose estimation system based on deep neural network and ICP registration for robotic spray painting application. Int J Adv Manuf Technol 104:285–299. https://doi.org/10.1007/s00170-019-03901-0

    Article  Google Scholar 

  11. Chen L, Zhu JM, Shi YY et al (2020) Identification method of surface roughness grade of parts based on deep learning. Agric Equip Veh Eng 58(10):76–79+122

    Google Scholar 

  12. Li ZY, Ma T, Li YB (2019) High-precision measurement method of circular pose in stereo vision space based on optimal projection plane[J]. Patt Recog Artif Intell 32(01):58–66

    Google Scholar 

  13. Wang H, Sridhar S, Huang J et al (2019) Normalized object coordinate space for category-level 6d object pose and size estimation.2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2019(29):2637–2646

  14. Li Q, Hu R, Xiao J et al (2020) Learning latent geometric consistency for 6D object pose estimation in heavily cluttered scenes. J Vis Commun Image Represent 10(2):790–801

    Google Scholar 

  15. Calonder M , Lepetit V, Strecha C, et al. (2010) BRIEF: Binary Robust Independent Elementary Features [C]. Computer Vision-ECCV 2010, 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV

  16. Soetedjo A, Yamada K (2005) Fast and robust traffic sign detection, 2005 IEEE International Conference on Systems, Man and Cybernetics, pp 1341–1346. https://doi.org/10.1109/ICSMC.2005.1571333

  17. Yoo JC, Han TH (2009) Fast normalized cross-correlation. Circ Syst Signal Process 28(6):819–832

    Article  MATH  Google Scholar 

  18. Gavrila DM, DM Gavrila V (1999) Real-time object detection for smart vehicles. IEEE Int Conf Comput Vision 1:87–93

    Google Scholar 

  19. M. Fischler, R Bolles. (1981) Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography [J]. CACM. 24(6):381–395

  20. Böhnke K, Gottscheber A (2010) Fast object registration and robotic bin picking. Springer, Berlin, Heidelberg, pp 23–37

    Google Scholar 

  21. Lee S, Kim J, Lee M, Yoo K, Barajas LG, Menassa R (2012) 3D visual perception system for bin picking in automotive subassembly automation, in 2012 IEEE International Conference on Automation Science and Engineering (CASE), pp 706–713

  22. Oh J-K, Lee S, Lee C-H (2012) Stereo vision based automation for a bin-picking solution. Int J Control Autom Syst 10(2):362–373

    Article  Google Scholar 

  23. Lei P, Zheng L, Xiao W, Li C, Wang D (2017) A closed-loop machining system for assembly interfaces of large-scale component based on extended STEP-NC. Int J Adv Manuf Technol 91:2499–2525

    Article  Google Scholar 

  24. Fang Z, Xia L, Chen G, Huang Y, Xu D, Tan M (2014) Vision-based alignment control for grating tiling in petawatt-class laser system. IEEE Trans Instrum Meas 63(6):1628–1638

    Article  Google Scholar 

  25. Liu S, Xu D, Liu F, Zhang D, Zhang Z (2016) Relative pose estimation for alignment of long cylindrical components based on microscopic vision. IEEE Trans Mechatron 21(3):1388–1398

    Article  Google Scholar 

  26. Shen Y, Wan W, Lu H (2017) Automatic sample alignment under microscopy for 360◦ imaging based on the nanorobotic manipulation system. IEEE Trans Robot 33(1):220–226

    Article  Google Scholar 

  27. Wang J, Cho H (2008) Micropeg and hole alignment using image moments based visual servoing method. IEEE Trans Ind Electron 55(3):1286–1294

    Article  Google Scholar 

  28. Liu S, Xu D, Zhang D, Zhang Z (2016) High precision automatic assembly based on microscopic vision and force information. IEEE Trans Autom Sci Eng 13(1):382–393

    Article  Google Scholar 

  29. Djordjevic V, Stojanovic V, Tao H, Song X, He S, Gao W (2021) Data-driven control of hydraulic servo actuator based on adaptive dynamic programming. Discret Contin Dyn Syst-S

  30. Stojanovic V, Nedic N (2016) Joint state and parameter robust estimation of stochastic nonlinear systems. Int J Robust Nonlinear Control 26:3058–3074

    Article  MathSciNet  MATH  Google Scholar 

  31. Nedic N, Stojanovic V, Djordjevic V (2015) Optimal control of hydraulically driven parallel robot platform based on firefly algorithm. Nonlinear Dyn 82:1457–1473

    Article  MathSciNet  MATH  Google Scholar 

  32. Tao HF, Long C, Qiu J, Stojanovic V (2022) Few shot cross equipment fault diagnosis method based on parameter optimization and feature mertic. Meas Sci Technol 33:115005

    Article  Google Scholar 

  33. Zhuang Z, Tao H, Chen Y, Stojanovic V, Paszke W (2022) Iterative learning control for repetitive tasks with randomly varying trial lengths using successive projection. Int J Adapt Control Signal Process 36(5):1196–1215

    Article  MathSciNet  Google Scholar 

  34. Girshick R, Donahue J, Darrell T et al (2014) Rich feature hierardries for accurate object detection and semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE, 580–587

  35. Girshick R. Fast R-CNN (2015) Proceedings of the IEEE International Conference on Computer Vision. Los Alamitos: IEEE, 1440–1448.

  36. RenSQ HKM, Girshick R et al (2017) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149

    Article  Google Scholar 

  37. Peng H, Xiao SH, Ruan JH et al (2020) (2020) Research on surface defect detection of particleboard based on faster R-CNN. Modular Mach Tools Autom Manuf Technol 03:91–94

    Google Scholar 

  38. Simo N, Yan K, Zisser Man A (2014) Very deep convolutional networks for large-scale image recognition[J]. Comput Sci 1409(15):1556–1563

    Google Scholar 

  39. He KM, Zhang XY, Ren SQ, et al. (2016) Deep residual learning for image recognition [C]. Proceedings of the lEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2016:770–778

  40. Zheng JL (2001) Signals and systems. Higher Education Press, Beijing

    Google Scholar 

  41. MITRA SK (2012) Digital signal processing: a computer-based approach. Electronic Industry Press, Beijing

    Google Scholar 

  42. Kuglin CD, Hines DC (1975) The phase correlation image alignment method [C]. Proceedings of IEEE International Conference on Cybernetics and Society. IEEE. 1975:163–165

Download references

Funding

The authors wish to express their gratitude to the financial support from the National Natural Science Foundation of China (No. 51975213).

Author information

Authors and Affiliations

  1. College of Mechanical and Power Engineering, East China University of Science and Technology, Shanghai, 200237, China

    Zhou Yibang, Wang Xiaoyong & Zhang Lanzhu

Authors
  1. Zhou Yibang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wang Xiaoyong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhang Lanzhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Zhou Yibang was a major contributor in writing the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Zhang Lanzhu.

Ethics declarations

Competing interests

The authors declare no competing interests.

Open access

This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yibang, Z., Xiaoyong, W. & Lanzhu, Z. Visual identification and pose estimation algorithms of nut tightening robot system. Int J Adv Manuf Technol 127, 5307–5326 (2023). https://doi.org/10.1007/s00170-023-11597-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-023-11597-6

Keywords

Search

Navigation