Skip to main content
SpringerLink
Log in

Seeing is believing: AR-assisted blind area assembly to support hand–eye coordination

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

The assembly stage is a vital phase in the production process and currently, there are still many manual tasks in the assembly operation. One of the challenges of manual assembly is the issue of blind area assembly since the visual obstruction of the hands or a part can lead to more errors and lower assembly efficiency. In this study, we developed an AR-assisted assembly system that solves the occlusion problem. Assembly workers can use the system to achieve comprehensive and precise hand–eye coordination (HEC). Additionally, we designed and conducted a user evaluation experiment to measure the learnability, usability, and mental effort required for the system for other HEC modes. Results indicate that hand position is the first visual information that should be considered in blind areas. Besides, the Intact HEC mode can effectively reduce the difficulty of learning and mental burden in operation, while at the same time improving efficiency.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

Data from user experiment are submitted as additional material. The problems of user experiment are discussed in this paper.

Code availability

The code of this paper is available.

References

  1. Wang X, Ong SK, Nee AYC (2016) A comprehensive survey of augmented reality assembly research. Advances in Manufacturing 1:1–22

    Article  Google Scholar 

  2. Seki H, Nose T, Lee YH, Chien CF, Gen M (2013) Special issue on recent advances in intelligent manufacturing systems. Comput Ind Eng 65(1):1

    Article  Google Scholar 

  3. Oztemel E, Gursev S (2018) Literature review of Industry 4.0 and related technologies. J Intell Manuf 47(4):1–56

  4. Li L (2018) China’s manufacturing locus in 2025: with a comparison of “Made-in-China. 2025” and “Industry 4.0.” Technol Forecast Soc Chang 135:66–74

    Article  Google Scholar 

  5. Egger J, Masood T (2020) Augmented reality in support of intelligent manufacturing–a systematic literature review. Comput Ind Eng 140:106195

  6. Chu CH, Ko CH (2021) An experimental study on augmented reality assisted manual assembly with occluded components. J Manuf Syst

  7. MacAllister A, Hoover M, Gilbert S, Oliver J, Radkowski R, Garrett T, Holub J, Winer E, Terry S, Davies P (2017) Comparing visual assembly aids for augmented reality work instructions. Proceedings of the Inter-service/Industry Training, Simulation, and Education Conference (I/ITSEC)

  8. Tang A, Owen C, Biocca F, Mou W (2003) Comparative effectiveness of augmented reality in object assembly. ACM Proceedings of the SIGCHI Conference on Human Factors in Computing Systems 2003:73–80

    Article  Google Scholar 

  9. Blattgerste J, Strenge B, Renner P, Pfeiffer T, Essig K (2017) Comparing conventional and augmented reality instructions for manual assembly tasks. Proceedings of the 10th International Conference on Pervasive Technologies Related to Assistive Environments 75–82

  10. Boothroyd G (1994) Product design for manufacture and assembly. Comput Aided Des 26(7):505–520

    Article  Google Scholar 

  11. Azuma RT (1997) A survey of augmented reality. Presence: Teleoperators & Virtual Environments 6(4):355–385

  12. Billinghurst M, Clark A, Lee G (2015) A survey of augmented reality. Foundations and Trends in Human-Computer Interaction 8(2–3):73–272

    Article  Google Scholar 

  13. Friedrich W (2002) ARVIKA — augmented reality for development, production and service. In: Mixed and augmented reality, 2002. In ISMAR 2002 

  14. Yao X, Zhou J, Lin Y, Li Y, Yu H, Lui Y (2017) Smart manufacturing based on cyber-physical systems and beyond. J Intell Manuf 10(1):175

    Google Scholar 

  15. Nee AY, Ong SK, Chryssolouris G, Mourtzis D (2012) Augmented reality applications in design and manufacturing. CIRP Annals 61(2)

  16. Malik AA, Masood T, Bilberg A (2019) Virtual reality in manufacturing: immersive and collaborative artificial-reality in design of human-robot workspace. Int J Comput Integr Manuf 1

  17. Baroroh DK, Chu CH, Wang L (2020) Systematic literature review on augmented reality in smart manufacturing: collaboration between human and artificial intelligence. J Manuf Syst

  18. Zhang S, He W, Wang S, Feng S, Hou Z, Hu Y (2021) An AR-enabled see-through system for vision blind areas. 23RD International Conference on Human-Computer Interaction

  19. Mahmoud N, Grasa SG, Nicolau SA, Doignon C, Soler L, Marescaux J, Montiel JMM (2017) On-patient see-through augmented reality based on visual SLAM. Int J Comput Assist Radiol Surg 12(1):1–11

    Article  Google Scholar 

  20. Wu ML, Chien JC, Wu CT, Lee JD (2018) An augmented reality system using improved-iterative closest point algorithm for on-patient medical image visualization. Sensors 18(8):2505

    Article  Google Scholar 

  21. Gupta S, Chaudhary R, Gupta S, Kaur A, Mantri A (2019) A survey on tracking techniques in augmented reality based application. In: ICIIP (International Conference on Image Information Processing)

  22. Thomas PC, David WM (1992) Augmented reality: an application of heads-up display technology to manual manufacturing processes. In International Conference on System Sciences

  23. Rios H, González E, Rodriguez CA, Siller HR, Contero M (2013) A mobile solution to enhance training and execution of troubleshooting techniques of the engine air bleed system on Boeing 737. Procedia Comput Sci 25:161–170

    Article  Google Scholar 

  24. Serván J, Mas F, Menéndez JL, Ríos J (2012) Using augmented reality in AIRBUS A400M shop floor assembly work instructions. In: AIP Conference Proceedings - 4th Manufacturing Eng Society Intl Conf - MESIC 2011

  25. Henderson S, Feiner S (2011) Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE transactions on visualization computer graphics

  26. Wang Z, Zhang S, Bai X (2019) Augmented reality based product blind area assembly assistance. Manufacturing Automation 41(1):29–35

    Google Scholar 

  27. Johansson RS, Westling G, Bäckström A, Flanagan JR (2001) Eye–hand coordination in object manipulation. J Neurosci 21(17):6917–6932

  28. Battaglia-Mayer A, Caminiti R (2018) Parieto-frontal networks for eye–hand coordination and movements - sciencedirect. Handb Clin Neurol 151:499–524

    Article  Google Scholar 

  29. Crawford JD, Medendorp WP, Marotta JJ (2004) Spatial transformations for eye–hand coordination. J Neurophysiol 92

  30. Siong LC (2015) Training and assessment of hand-eye coordination with electroencephalography

  31. Chen CC, Chou CH, Ciou SH, Hsieh WM, Hong DJ, Wang CH et al (2014) Interactive technology for the elderly in the hand-eye coordination training. Appl Mech Mater 696:281–286

    Article  Google Scholar 

  32. Wang Y, Zhang S, Bai X (2019) A 3D tracking and registration method based on point cloud and visual features for augmented reality aided assembly system. Journal of Northwestern Polytechnical University 37(1):143–151

    Article  Google Scholar 

  33. Leap Motion https://www.ultraleap.com/

  34. Vofuria https://developer.vuforia.com/

  35. Bangor A, Kortum PT, Miller JT (2009) Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 4(3):114–123

    Google Scholar 

  36. Olivier R, Hullenaar C, Ruurda JP, Broeders I (2009) Ergonomics, user comfort, and performance in standard and robot-assisted laparoscopic surgery. Surg Endosc 23(6):1365–1371

    Article  Google Scholar 

  37. Tang YM, Au KM, Leung Y (2018) Comprehending products with mixed reality: geometric relationships and creativity. Int J Eng Bus 10

  38. Hololens https://www.microsoft.com/en-us/hololens/hardware

  39. Wang P, Zhang S, Bai X, Billinghurst M, He W, Sun M, Chen Y, Lv H, Ji H (2019) 2.5dhands: a gesture-based mr remote collaborative platform. Int J Adv Manuf Syst

  40. Andrade C (2019) The P value and statistical significance: misunderstandings, explanations, challenges, and alternatives. Indian J Psychol Med 41(3):210–215

    Article  Google Scholar 

Download references

Funding

This study was funded by the National Key R&D Program of China (Grant No. 2019YFB1703800).

Author information

Authors and Affiliations

  1. Cyber-Physical Interaction Lab, School of Mechanical Engineering, Northwestern Polytechnical University, Xi’an, Shaanxi Province, China

    Shuo Feng, Weiping He, Shaohua Zhang & Mark Billinghurst

  2. Empathic Computing Lab, University of South Australia, Adelaide, Australia

    Mark Billinghurst

Authors
  1. Shuo Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weiping He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shaohua Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mark Billinghurst
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Shuo Feng is the first author of this paper. Weiping He and Mark Billinghurst Participated in the writing of the article. Shaohua Zhang assisted to complete the system construction and user experiments.

Corresponding authors

Correspondence to Shuo Feng or Weiping He.

Ethics declarations

Conflict of interest

The authors have no relevant financial to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (XLSX 11 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Feng, S., He, W., Zhang, S. et al. Seeing is believing: AR-assisted blind area assembly to support hand–eye coordination. Int J Adv Manuf Technol 119, 8149–8158 (2022). https://doi.org/10.1007/s00170-021-08546-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-021-08546-6

Keywords

Search

Navigation