Skip to main content
SpringerLink
Log in

A marker-less monocular vision point positioning method for industrial manual operation environments

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

Vision-assisted technologies in industrial manual operation such as augmented reality (AR) are increasingly popular. They require high positioning accuracy and robustness to operate normally. However, narrow spaces, moving hands, or tools may occlude or obscure local visual features of operation environments and thus affect the positioning accuracy and robustness of operating position. The resultant misguidance may even cause misoperation of operators. This paper proposes a marker-less monocular vision point positioning method for vision-assisted manual operation in industrial environments. The proposed method can accurately and robustly locate the target point of operation using constraint minimization method even if the target area has no corresponding visual features in the case of occlusion and improper illumination. The proposed method has three phases: intersection generation, intersection optimization, and target point solving. In the intersection generation phase, a certain number of intersections of epipolar lines are generated as candidate target points using fundamental matrices. Here, the solving constraint is converted from point-to-line to point-to-points. In the intersection optimization phase, the intersections are optimized to two different sets through the iterative linear fitting and geometric mean absolute error methods. Here, the solving constraint is further converted from point-to-points to point-to-point sets. In the target point solving phase, the target point is solved as a constrained minimization problem based on the distribution constraint of the two intersection sets. Here, the solving constraint is finally converted from point-to-point sets to point-to-point, and the unique optimal solution is obtained as the target point. The experimental results show that this method has a better accuracy and robustness than the traditional homography matrix method for the practical industrial operation scenes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data availability

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

Code availability

Not applicable.

References

  1. Lopik KV, Sinclair M, Sharpe R, Conway P, West A (2020) Developing augmented reality capabilities for industry 4.0 small enterprises: lessons learnt from a content authoring case study. Comput Indust 117(2020): 103208. https://doi.org/10.1016/j.compind.2020.103208

  2. Bottani E, Vignali G (2019) Augmented reality technology in the manufacturing industry: a review of the last decade. IISE Transactions 51(3):284–310. https://doi.org/10.1080/24725854.2018.1493244

    Article  Google Scholar 

  3. Palmarini R, Erkoyuncu JA, Roy R, Torabmostaedi H (2018) A systematic review of augmented reality applications in maintenance. Robot Comput Integr Manuf 49(2018):215–228. https://doi.org/10.1016/j.rcim.2017.06.002

    Article  Google Scholar 

  4. Wang X, Ong SK, Nee AYC (2016) A comprehensive survey of augmented reality assembly research. Adv Manuf 4(1):1–22. https://doi.org/10.1007/s40436-015-0131-4

    Article  Google Scholar 

  5. Nee AYC, Ong SK, Chryssolouris G, Mourtzis D (2012) Augmented reality applications in design and manufacturing. CIRP Ann 61(2):657–679. https://doi.org/10.1016/j.cirp.2012.05.010

    Article  Google Scholar 

  6. Lima JP, Roberto R, Simões F, Almeida M, Figueiredo L, Teixeira JM, Teichrieb V (2017) Markerless tracking system for augmented reality in the automotive industry. Expert Syst Appl 82(2017):100–114. https://doi.org/10.1016/j.eswa.2017.03.060

    Article  Google Scholar 

  7. Siew CY, One SK, Nee AYC (2019) A practical augmented reality-assisted maintenance system framework for adaptive user support. Robot Comput Integr Manuf 59(2019):115–129. https://doi.org/10.1016/j.rcim.2019.03.010

    Article  Google Scholar 

  8. Radkowski R, Herrema J, Oliver J (2015) Augmented reality-based manual assembly support with visual features for different degrees of difficulty. Int J Human-Comput Int 31(5):337–349. https://doi.org/10.1080/10447318.2014.994194

    Article  Google Scholar 

  9. Henderson SJ, Feiner S (2011) Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE Trans Visual Comput Graphics 17(10):1355–1368. https://doi.org/10.1109/tvcg.2010.245

    Article  Google Scholar 

  10. Geng J, Tian X, Bai M, Jia X, Liu X (2014) A design method for three-dimensional maintenance, repair and overhaul job card of complex products. Comput Ind 65(1):200–209. https://doi.org/10.1016/j.compind.2013.08.008

    Article  Google Scholar 

  11. Gattullo M, Scurati GW, Fiorentino M, Uva AE, Ferrise F, Bordegoni M (2019) Towards augmented reality manuals for industry 4.0: a methodology. Robot Comput Integr Manuf 56 (2019):276–286. https://doi.org/10.1016/j.rcim.2018.10.001

  12. Tatić D, Tešić B (2017) The application of augmented reality technologies for the improvement of occupational safety in an industrial environment. Comput Ind 85(2017):1–10. https://doi.org/10.1016/j.compind.2016.11.004

    Article  Google Scholar 

  13. Scurati GW, Gattullo M, Fiorentino M, Ferrise F, Bordegoni M, Uva AE (2018) Converting maintenance actions into standard symbols for Augmented Reality applications in Industry 4.0. Comput Ind 98(2018):68–79. https://doi.org/10.1016/j.compind.2018.02.001

  14. Fox S (2010) The importance of information and communication design for manual skills instruction with augmented reality. J Manuf Technol Manag 21(2):188–205. https://doi.org/10.1108/17410381011014369

    Article  Google Scholar 

  15. Zubizarreta J, Aguinaga I, Amundarain A (2019) A framework for augmented reality guidance in industry. Int J Adv Manuf Technol 102(9):4095–4108. https://doi.org/10.1007/s00170-019-03527-2

    Article  Google Scholar 

  16. Wang Y, Zhang S, Wan B, He W, Bai X (2018) Point cloud and visual feature-based tracking method for an augmented reality-aided mechanical assembly system. Int J Adv Manuf Technol 99(9):2341–2352. https://doi.org/10.1007/s00170-018-2575-8

    Article  Google Scholar 

  17. Lu Y, Gao K, Zhang T, Xu T (2018) A novel image registration approach via combining local features and geometric invariants. PLoS One 13(1):e0190383–e0190383. https://doi.org/10.1371/journal.pone.0190383

    Article  Google Scholar 

  18. Schmalstieg D, Hollerer T (2016) Augmented Reality: principles and practice. Addison-Wesley Professional

    Google Scholar 

  19. Yazdi M, Bouwmans T (2018) New trends on moving object detection in video images captured by a moving camera: a survey. Comp Sci Rev 28(2018):157–177. https://doi.org/10.1016/j.cosrev.2018.03.001

    Article  MathSciNet  Google Scholar 

  20. Meshgi K, Ishii S (2015) The state-of-the-art in handling occlusions for visual object tracking. IEICE Trans Info Sys E98.D(7):1260–1274. https://doi.org/10.1587/transinf.2014EDR0002

  21. Bhaskar H, Dwivedi D, Dogra DP, Al-Mualla M, Mihaylova L (2015) Autonomous detection and tracking under illumination changes, occlusions and moving camera. Signal Process 117(2015):343–354. https://doi.org/10.1016/j.sigpro.2015.06.003

    Article  Google Scholar 

  22. Dey S, Reilly V, Saleemi I, Shah M (2012) Detection of independently moving objects in non-planar scenes via multi-frame monocular epipolar constraint. In: Fitzgibbon A., Lazebnik S., Perona P., Sato Y., Schmid C. (eds) Computer Vision – ECCV 2012. ECCV 2012. Lect Notes Comp Sci 7576. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33715-4_62

  23. Zhang Z (1998) Determining the epipolar geometry and its uncertainty: a review. Int J Comput Vision 27(2):161–195. https://doi.org/10.1023/A:1007941100561

    Article  Google Scholar 

  24. Szeliski R (2022) Computer vision: algorithms and applications. Springer Cham. https://doi.org/10.1007/978-3-030-34372-9

    Article  MATH  Google Scholar 

  25. Hartley R, Zisserman A (2004) Multiple View Geometry in Computer Vision (2nd ed.). Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511811685

  26. Ong SK, Yew AWW, Thanigaivel NK, Nee AYC (2020) Augmented reality-assisted robot programming system for industrial applications. Robot Comput Integr Manuf 61(2020)

  27. Xu J, Chen R, Chen H, Zhang S, Chen K (2017) Fast registration methodology for fastener assembly of large-scale structure. IEEE Trans Industr Electron 64(1):717–726. https://doi.org/10.1109/TIE.2016.2599140

    Article  Google Scholar 

  28. Xiao H, Duan Y, Zhang Z (2018) Mobile 3D assembly process information construction and transfer to the assembly station of complex products. Int J Comput Integr Manuf 31(1):11–26. https://doi.org/10.1080/0951192X.2017.1356470

    Article  Google Scholar 

  29. Li X, Yi W, Chi H, Wang X, Chan APC (2018) A critical review of virtual and augmented reality (VR/AR) applications in construction safety. Autom Constr 86(2018):150–162. https://doi.org/10.1016/j.autcon.2017.11.003

    Article  Google Scholar 

  30. Pan Z, Liu S, Fu W (2017) A review of visual moving target tracking. Multimedia Tools and Applications 76(16):16989–17018. https://doi.org/10.1007/s11042-016-3647-0

    Article  Google Scholar 

  31. Gao QH, Wan TR, Tang W, Chen L (2019) Object registration in semi-cluttered and partial-occluded scenes for augmented reality. Multimed Tools 78(11):15079–15099. https://doi.org/10.1007/s11042-018-6905-5

    Article  Google Scholar 

  32. Wang Y, Zhang S, Yang S, He W, Bai X, Zeng Y (2017) A LINE-MOD-based markerless tracking approachfor AR applications. Int J Adv Manuf Technol 89(5–8):1699–1707. https://doi.org/10.1007/s00170-016-9180-5

    Article  Google Scholar 

  33. Huang P, Cai J, Meng Z, Hu Z, Wang D (2014) Novel method of monocular real-time feature point tracking for tethered space robots. J Aerosp Eng 27(6):04014039. https://doi.org/10.1061/(ASCE)AS.1943-5525.0000367

    Article  Google Scholar 

  34. Wang K, Liu D, Duan G, Hu L, Tan J (2020) A fast object registration method for augmented reality assembly with simultaneous determination of multiple 2D–3D correspondences. Robot Comput Integr Manuf 63(2020)

  35. Goshtasby AA (2012) Image registration: principles, tools and methods. Springer, London. https://doi.org/10.1007/978-1-4471-2458-0

    Article  MATH  Google Scholar 

  36. Marchand E, Uchiyama H, Spindler F (2016) Pose estimation for augmented reality: a hands-on survey. IEEE Trans Visual Comput Graphics 22(12):2633–2651. https://doi.org/10.1109/tvcg.2015.2513408

    Article  Google Scholar 

  37. Maron H, Dym N, Kezurer I, Kovalsky S, Lipman Y (2016) Point registration via efficient convex relaxation. ACM Trans Graph 35:1–12. https://doi.org/10.1145/2897824.2925913

    Article  Google Scholar 

  38. Xiong Z, Zhang Y (2010) A critical review of image registration methods. Int J Image Data Fusion 1(2):137–158. https://doi.org/10.1080/19479831003802790

    Article  Google Scholar 

  39. Zhu H, Guo B, Zou K, Li Y, Yuen K, Mihaylova L, Leung H (2019) A review of point set registration: from pairwise registration to groupwise registration. Sensors 19(5):1191. https://doi.org/10.3390/s19051191

    Article  Google Scholar 

  40. Zhang Z, Deriche R, Faugeras O, Luong Q (1995) A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry. Artif Intell 78(1):87–119. https://doi.org/10.1016/0004-3702(95)00022-4

    Article  Google Scholar 

  41. Zamalieva D, Yilmaz A, Davis JW (2014) A multi-transformational model for background subtraction with moving cameras. In: Fleet D., Pajdla T., Schiele B., Tuytelaars T. (eds) Computer Vision – ECCV 2014. ECCV 2014. Lect Notes Comput Sci 8689. Springer, Cham. https://doi.org/10.1007/978-3-319-10590-1_52

  42. Ramachandran M, Veeraraghavan A, Chellappa R (2009) CHAPTER 5 - video stabilization and mosaicing, in The Essential Guide to Video Processing:109–140. https://doi.org/10.1016/B978-0-12-374456-2.00006-2

  43. Wang J, Gu G, Wan M, Ren K (2018) Dual-correlation transformation for image stitching. J Electron Imaging 27(3):033040. https://doi.org/10.1117/1.JEI.27.3.033040

    Article  Google Scholar 

  44. Fathy ME, Hussein AS, Tolba MF (2011) Fundamental matrix estimation: a study of error criteria. Pattern Recogn Lett 32(2):383–391. https://doi.org/10.1016/j.patrec.2010.09.019

    Article  Google Scholar 

  45. Basta T, Emirates U (2013) Does the fundamental matrix define a one-to-one relation between the corresponding image points of a scene? Int J Image Graph 1(3):125–128. https://doi.org/10.12720/JOIG.1.3.125-128

  46. Basta T (2014) Is the fundamental matrix really independent of the scene structure? Int J Signal Process, Image Processing and Pattern Recognition 7(5):149–167. https://doi.org/10.14257/ijsip.2014.7.5.13

  47. Stojanovic A, Unger M (2007) Robust detection of point correspondences in stereo images. Acta Polytechnica 47(4–5):23–28. https://doi.org/10.14311/956

  48. Ben-Artzi G, Werman M, Peleg S (2017) Camera calibration from dynamic silhouettes using motion barcodes. ArXiv 2017(Jan):1–9. https://doi.org/10.48550/arXiv.1506.07866

  49. Hassaballah M, Abdelmgeid AA, Alshazly HA (2016) Image features detection, description and matching, In: Awad A, Hassaballah M (eds) Image Feature Detectors and Descriptors. Stud Comput Intell 630. Springer, Cham. https://doi.org/10.1007/978-3-319-28854-3_2

  50. Stojanovic A, Unger M (2013) A new evaluation criterion for point correspondences in stereo images. In: Adami N, Cavallaro A, Leonardi R, Migliorati P (eds) Analysis, Retrieval and Delivery of Multimedia Content. Lect Notes Electr Eng 158. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-3831-1_11

Download references

Funding

This work was supported by the National Natural Science Foundation of China under Grant 52175484 and was largely conducted when the first author was attached at the University of Alberta. The corresponding author, Prof. Yongsheng Ma, had been supported by Canada NSERC Discovery grant; he has moved from University of Canada to Southern University of Science Technology in Shenzhen, China, since July, 2021.

Author information

Authors and Affiliations

  1. Institute of Intelligent Manufacturing, Northwestern Polytechnical University, Xi’an, 710072, China

    Junhao Geng, Xinyang Zhao & Zhenxin Guo

  2. AVIC XAC Commercial Aircraft Co., Ltd, Xi’an, 710089, China

    Shangan Zhang

  3. AVIC Chengdu Aircraft Industry (Group) Co., Ltd, Chengdu, 610092, China

    Jianjun Tang

  4. Department of Mechanical and Energy Engineering, Southern University of Science and Technology, Shenzhen, China

    Yiming Rong & Yongsheng Ma

  5. Department of Mechanical Engineering, Faculty of Engineering, University of Alberta, Edmonton, T6G 1H9, Canada

    Yongsheng Ma

Authors
  1. Junhao Geng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xinyang Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhenxin Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shangan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jianjun Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yiming Rong
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yongsheng Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All the authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by Junhao Geng, Xinyang Zhao, Zhenxin Guo, Shangan Zhang, Jianjun Tang, and Yiming Rong. The first draft of the manuscript was written by Junhao Geng and Yongsheng Ma, and all the authors commented on previous versions of the manuscript. All the authors read and approved the final manuscript.

Corresponding author

Correspondence to Yongsheng Ma.

Ethics declarations

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Geng, J., Zhao, X., Guo, Z. et al. A marker-less monocular vision point positioning method for industrial manual operation environments. Int J Adv Manuf Technol 120, 6011–6027 (2022). https://doi.org/10.1007/s00170-022-09123-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-022-09123-1

Keywords

Search

Navigation