Skip to main content

Advertisement

SpringerLink
Log in

Cross-platform AR annotation for assembly-design communication in pipe outfitting

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

Communication is an essential part of most professional activities. For complex industrial products such as ships, some design problems can only be found during the construction process. However, it is time-consuming for on-site assemblers to provide feedback on problems through traditional methods such as phone calls and photos. In recent years, adding structured visual cues through augmented reality (AR) has become an important method to assist remote collaborative tasks. However, previous studies and commercial solutions often had limited annotation types and relied on specific devices for tracking and reconstruction, which is unsatisfactory for deployment in industrial equipment and large scenes. The presented work tackles issues with feedback and collaborative decision-making of design problems in the pipe outfitting stage and proposes a cross-platform AR annotation solution on desktop computers, industrial tablets, and AR head-mounted devices (HMD). The novelty of the current work lies in a cross-functional annotation taxonomy for real-time collaboration, device-cloud integrative context awareness methods based on RGB-D sensors compatible with more devices, 2D–3D annotation mapping algorithms, and visual enhancement strategies for large scenes. A prototype system is developed and verified on a ship unit model and a physical pipe platform. The results indicate that the solution provides good performance in real-time capability and reconstruction precision, allows rapid localization of faulty parts, and creates annotations flexibly and reliably. This solution is also expected to play an active role in collaborative quality inspection and intelligent management in shipbuilding and other heavy industries.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Availability of data and material

Not applicable.

Code availability

Not applicable.

Abbreviations

AR:

Augmented reality

ARA:

Augmented reality annotation

ASQ:

After-scenario questionnaire

CAD:

Computer-aided design

CAR:

Collaborative augmented reality

CSCW:

Computer-supported cooperative work

HHD:

Hand-held device

HMD:

Head-mounted device

SAR:

Spatial augmented reality

SUS:

System usability scale

VR:

Virtual reality

References

  1. Zhang Q, Yue W, Wang C (2008) Research on the application of CSCW in shipbuilding industry. In: 2008 Int Symp Electronic Commerce Security 834–837

  2. Zhao DG, Jiang Y, Bao JW, Wang JQ, Jia H (2019) Study on batching and picking optimization of marine outfitting pallets. MATEC Web Conf 272:01015. https://doi.org/10.1051/matecconf/201927201015

    Article  Google Scholar 

  3. Molland AF (2008) Chapter 9 - Ship design, construction and operation. The Maritime Engineering Reference Book. Butterworth-Heinemann, Oxford, pp 636–727

    Google Scholar 

  4. Wu D, Yang R, Ma D, Fan X (2008) Integrated virtual assembly environment and its application in ship piping layout. Int J Prod Res 46:4729–4749. https://doi.org/10.1080/00207540601008424

    Article  MATH  Google Scholar 

  5. Kim H, Lee G-H (2020) Study on system support for offshore plant piping process using 3D simulator. J Ocean Eng Technol 34:217–226. https://doi.org/10.26748/KSOE.2019.102

  6. Jang M, Nam J-H (2020) Determination and application of installation sequence of piping systems in cramped spaces of ships and offshore structures considering geometric relationship of pipe elements. Int J Nav Archit Ocean Eng 12:60–70. https://doi.org/10.1016/j.ijnaoe.2019.07.001

    Article  Google Scholar 

  7. Alexopoulos K, Makris S, Xanthakis V, Sipsas K, Chryssolouris G (2016) A concept for context-aware computing in manufacturing: the white goods case. Int J Comput Integr Manuf 29:839–849. https://doi.org/10.1080/0951192X.2015.1130257

    Article  Google Scholar 

  8. Dai F, Olorunfemi A, Peng W, Cao D, Luo X (2021) Can mixed reality enhance safety communication on construction sites? An industry perspective Saf Sci 133:105009. https://doi.org/10.1016/j.ssci.2020.105009

    Article  Google Scholar 

  9. Aschenbrenner D, Rojkov M, Leutert F, Verlinden J, Lukosch S, Latoschik ME, Schilling K (2018) Comparing different augmented reality support applications for cooperative repair of an industrial robot. In: 2018 IEEE Int Symp Mixed Augmented Reality Adjunct (ISMAR-Adjunct) 69–74

  10. Greenfield MJ, Luck J, Billingsley ML, Heyes R, Smith OJ, Mosahebi A, Khoussa A, Abu-Sittah G, Hachach-Haram N (2018) Demonstration of the effectiveness of augmented reality telesurgery in complex hand reconstruction in Gaza. Plast Reconstr Surg – Glob Open 6:e1708. https://doi.org/10.1097/GOX.0000000000001708

  11. Wang P, Bai X, Billinghurst M, Zhang S, Wei S, Xu G, He W, Zhang X, Zhang J (2020) 3DGAM: using 3D gesture and CAD models for training on mixedreality remote collaboration. Multimed Tools Appl. https://doi.org/10.1007/s11042-020-09731-7

    Article  Google Scholar 

  12. Madeira T, Marques B, Alves J, Dias P, Santos BS (2021) Exploring annotations and hand tracking in augmented reality for remote collaboration. In: Karwowski W, Ahram T, Etinger D, Tanković N, Taiar R (eds) Human Systems Engineering and Design III. Springer International Publishing, Cham, pp 83–89

    Chapter  Google Scholar 

  13. Fang D, Xu H, Yang X, Bian M (2020) An augmented reality-based method for remote collaborative real-time assistance: from a system perspective. Mob Netw Appl 25:412–425. https://doi.org/10.1007/s11036-019-01244-4

    Article  Google Scholar 

  14. Riedlinger U, Oppermann L, Prinz W (2019) Tango vs. HoloLens: a comparison of collaborative indoor AR visualisations using hand-held and hands-free devices. Multimodal Technol Interact 3:23. https://doi.org/10.3390/mti3020023

  15. Calandra D, Cannavò A, Lamberti F (2021) Improving AR-powered remote assistance: a new approach aimed to foster operator’s autonomy and optimize the use of skilled resources. Int J Adv Manuf Technol 114:3147–3164

  16. Build new augmented reality experiences that seamlessly blend the digital and physical worlds. In: Google Dev. https://developers.google.com/ar. Accessed 16 Aug 2021

  17. Augmented reality. In: Apple Dev. https://developer.apple.com/augmented-reality/. Accessed 16 Aug 2021

  18. Mourtzis D, Siatras V, Angelopoulos J (2020) Real-time remote maintenance support based on augmented reality (AR). Appl Sci 10:1855. https://doi.org/10.3390/app10051855

    Article  Google Scholar 

  19. Liu F, Li Y, Hao R (2019) Design and implementation of a collaborative design system for ships. J Coast Res 98:104–109. https://doi.org/10.2112/SI98-026.1

    Article  Google Scholar 

  20. Li X, Roh M-I, Ham S-H (2019) A collaborative simulation in shipbuilding and the offshore installation based on the integration of the dynamic analysis, virtual reality, and control devices. Int J Nav Archit Ocean Eng 11:699–722

  21. Vidal-Balea A, Blanco-Novoa O, Fraga-Lamas P, Vilar-Montesinos M, Fernández-Caramés TM (2020) Creating collaborative augmented reality experiences for Industry 4.0 training and assistance applications: performance evaluation in the shipyard of the future. Appl Sci 10:9073. https://doi.org/10.3390/app10249073

  22. Cascini G, O’Hare J, Dekoninck E, Becattini N, Boujut J-F, Ben Guefrache F, Carli I, Caruso G, Giunta L, Morosi F (2020) Exploring the use of AR technology for co-creative product and packaging design. Comput Ind 123:103308. https://doi.org/10.1016/j.compind.2020.103308

    Article  Google Scholar 

  23. Wang J, Qi Y (2022) A multi-user collaborative AR system for industrial applications. Sensors 22:1319. https://doi.org/10.3390/s22041319

    Article  Google Scholar 

  24. Liu X, Zheng L, Shuai J, Zhang R, Li Y (2020) Data-driven and AR assisted intelligent collaborative assembly system for large-scale complex products. Procedia CIRP 93:1049–1054. https://doi.org/10.1016/j.procir.2020.04.041

    Article  Google Scholar 

  25. Wang Z, Zhang S, Bai X (2021) A mixed reality platform for assembly assistance based on gaze interaction in industry. Int J Adv Manuf Technol 116:3193–3205. https://doi.org/10.1007/s00170-021-07624-z

    Article  Google Scholar 

  26. García-Pereira I, Portalés C, Gimeno J, Casas S (2020) A collaborative augmented reality annotation tool for the inspection of prefabricated buildings. Multimed Tools Appl 79:6483–6501. https://doi.org/10.1007/s11042-019-08419-x

    Article  Google Scholar 

  27. Marino E, Barbieri L, Colacino B, Fleri AK, Bruno F (2021) An augmented reality inspection tool to support workers in Industry 4.0 environments. Comput Ind 127:103412. https://doi.org/10.1016/j.compind.2021.103412

  28. Choi SH, Kim M, Lee JY (2018) Situation-dependent remote AR collaborations: image-based collaboration using a 3D perspective map and live video-based collaboration with a synchronized VR mode. Comput Ind 101:51–66. https://doi.org/10.1016/j.compind.2018.06.006

    Article  Google Scholar 

  29. Bruno F, Barbieri L, Marino E, Muzzupappa M, D’Oriano L, Colacino B (2019) An augmented reality tool to detect and annotate design variations in an Industry 4.0 approach. Int J Adv Manuf Technol 105:875–887. https://doi.org/10.1007/s00170-019-04254-4

    Article  Google Scholar 

  30. Fernández del Amo I, Erkoyuncu J, Vrabič R, Frayssinet R, Vazquez Reynel C, Roy R (2020) Structured authoring for AR-based communication to enhance efficiency in remote diagnosis for complex equipment. Adv Eng Inform 45:101096. https://doi.org/10.1016/j.aei.2020.101096

    Article  Google Scholar 

  31. Marques B, Teixeira A, Silva S, Alves J, Dias P, Santos BS (2022) A critical analysis on remote collaboration mediated by augmented reality: making a case for improved characterization and evaluation of the collaborative process. Comput Graph 102:619–633. https://doi.org/10.1016/j.cag.2021.08.006

    Article  Google Scholar 

  32. Hansen FA (2006) Ubiquitous annotation systems: technologies and challenges. In: Proceedings of the seventeenth conference on Hypertext and hypermedia - HYPERTEXT ’06. ACM Press, Odense, Denmark, pp 121

  33. Gauglitz S, Nuernberger B, Turk M, Höllerer T (2014) In touch with the remote world: remote collaboration with augmented reality drawings and virtual navigation. In: Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology. Association for Computing Machinery, New York, NY, USA, pp 197–205

  34. Wither J, DiVerdi S, Höllerer T (2009) Annotation in outdoor augmented reality. Comput Graph 33:679–689. https://doi.org/10.1016/j.cag.2009.06.001

    Article  Google Scholar 

  35. Keil J, Schmitt F, Engelke T, Graf H, Olbrich M (2018) Augmented reality views: discussing the utility of visual elements by mediation means in industrial AR from a design perspective. Virtual, augmented and mixed reality: applications in health, cultural heritage, and industry. Springer, Cham, pp 298–312

    Google Scholar 

  36. García-Pereira I, Gimeno J, Morillo P, Casanova-Salas P (2020) A taxonomy of augmented reality annotations: In: Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications. SCITEPRESS - Science and Technology Publications, Valletta, Malta, pp 412–419

  37. Nuernberger B, Lien K, Höllerer T, Turk M (2016) Interpreting 2D gesture annotations in 3D augmented reality. In: 2016 IEEE Symp 3D User Interfaces (3DUI) 149–158

  38. Chang YS, Nuernberger B, Luan B, Höllerer T, O’Donovan J (2017) Gesture-based augmented reality annotation. In: 2017 IEEE Virtual Reality (VR) 469–470

  39. Cao D, Kim S (2020) Immediate presence of handwritten annotation on moving picture in real-time collaborative system. IEEE Access 8:133258–133268. https://doi.org/10.1109/ACCESS.2020.3010639

    Article  Google Scholar 

  40. Fakourfar O, Ta K, Tang R, Bateman S, Tang A (2016) Stabilized annotations for mobile remote assistance. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, pp 1548–1560

  41. Zhang Y, Zi Y (2020) Mixed reality annotations system for museum space based on the UWB positioning and mobile device. In: De Paolis LT, Bourdot P (eds) Augmented reality, virtual reality, and computer graphics. Springer International Publishing, Cham, pp 328–342

    Chapter  Google Scholar 

  42. Egodagamage R, Tuceryan M (2018) Distributed monocular visual SLAM as a basis for a collaborative augmented reality framework. Comput Graph 71:113–123. https://doi.org/10.1016/j.cag.2018.01.002

    Article  Google Scholar 

  43. Campos C, Elvira R, Rodríguez JJG, M. Montiel JM, D. Tardós J, (2021) ORB-SLAM3: an accurate open-source library for visual, visual–inertial, and multimap SLAM. IEEE Trans Robot. https://doi.org/10.1109/TRO.2021.3075644

    Article  Google Scholar 

  44. Rosinol A, Violette A, Abate M, Hughes N, Chang Y, Shi J, Gupta A, Carlone L (2021) Kimera: from SLAM to spatial perception with 3D dynamic scene graphs. ArXiv210106894 Cs

  45. Mohr P, Mori S, Langlotz T, Thomas BH, Schmalstieg D, Kalkofen D (2020) Mixed reality light fields for interactive remote assistance. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, pp 1–12

  46. Vuforia Chalk Augmented Reality (AR) Remote assistance | PTC. https://www.ptc.com/en/products/vuforia/vuforia-chalk. Accessed 27 Dec 2020

  47. Enterprise augmented reality remote assistance & support solutions | Scope AR. https://www.scopear.com/home/. Accessed 11 Dec 2021

  48. Strasdat H, Davison AJ, Montiel JMM, Konolige K (2011) Double window optimisation for constant time visual SLAM. 2011 International Conference on Computer Vision. IEEE, Barcelona, pp 2352–2359

    Chapter  Google Scholar 

  49. Kazhdan M, Chuang M, Rusinkiewicz S, Hoppe H (2020) Poisson surface reconstruction with envelope constraints. Comput Graph Forum 39:173–182. https://doi.org/10.1111/cgf.14077

    Article  Google Scholar 

  50. Werner D, Al-Hamadi A, Werner P (2014) Truncated signed distance function: experiments on voxel size. In: Campilho A, Kamel M (eds) Image analysis and recognition. Springer International Publishing, Cham, pp 357–364

    Chapter  Google Scholar 

  51. Labbé M, Michaud F (2019) RTAB-Map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation. J Field Robot 36:416–446. https://doi.org/10.1002/rob.21831

    Article  Google Scholar 

  52. Marques B, Silva S, Teixeira A, Dias P, Santos BS (2022) A vision for contextualized evaluation of remote collaboration supported by AR. Comput Graph 102:413–425. https://doi.org/10.1016/j.cag.2021.10.009

    Article  Google Scholar 

  53. Lewis JR (1991) Psychometric evaluation of an after-scenario questionnaire for computer usability studies: the ASQ. ACM SIGCHI Bull 23:78–81. https://doi.org/10.1145/122672.122692

    Article  Google Scholar 

  54. Brooke J (1996) SUS: a quick and dirty usability scale. In: Usability evaluation in industry. CRC Press

  55. Lewis JR (2018) The system usability scale: past, present, and future. Int J Human-Computer Interact 34:577–590. https://doi.org/10.1080/10447318.2018.1455307

    Article  Google Scholar 

  56. Sauro J, Lewis JR (2016) Chapter 8 - Standardized usability questionnaires. In: Sauro J, Lewis JR (eds) Quantifying the User Experience, 2nd edn. Morgan Kaufmann, Boston, pp 185–248

    Chapter  Google Scholar 

Download references

Funding

This work was supported by the Ministry of Industry and Information Technology of the People’s Republic of China (Grant No. CJ04N20) and National Natural Science Foundation of China (Grant No. 51975362).

Author information

Authors and Affiliations

  1. Institute of Intelligent Manufacturing and Information Engineering, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China

    Jinge Wang, Xiumin Fan, Xu Yang & Xuyue Yin

  2. Jiangnan Institute of Technology, Jiangnan Shipyard (Group) Co., Ltd, Shanghai, 201913, China

    Yu Zhu

Authors
  1. Jinge Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiumin Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yu Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xu Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xuyue Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Jinge Wang, methodology, software, conceptualization, formal analysis, and writing — original draft. Xiumin Fan, conceptualization, supervision, and writing — review. Yu Zhu, resources and validation. Xu Yang, software and validation. Xuyue Yin, writing — review and editing.

Corresponding author

Correspondence to Xiumin Fan.

Ethics declarations

Ethics approval

This work is in compliance with the ethical standards.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, J., Fan, X., Zhu, Y. et al. Cross-platform AR annotation for assembly-design communication in pipe outfitting. Int J Adv Manuf Technol 121, 4981–4998 (2022). https://doi.org/10.1007/s00170-022-09637-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-022-09637-8

Keywords

Search

Navigation