Skip to main content

Advertisement

Log in

Remote collaboration in maintenance contexts using augmented reality: insights from a participatory process

  • Industrial Paper
  • Published:
International Journal on Interactive Design and Manufacturing (IJIDeM) Aims and scope Submit manuscript

Abstract

Problem solving in Industry 4.0 often requires collaboration among remote team members, which face increased complexity on their daily tasks and require mechanisms with adaptive capabilities to share and combine knowledge. Augmented Reality (AR) is one of the most promising solutions, allowing taking advantage from seamless integration of virtual and real-world objects, which can be used to provide a shared understanding of the task and context. In this regard, most research works, so far, have been devoted to explore and evolve the necessary technology. However, it is now important to revisit the subject of remote collaboration in relation with AR to understand how much of the collaborative effort can already be supported and identify gaps that should inform further research. In line with this mindset, we adopted a user-centered approach with partners from the industry sector, including participatory design and a focus group with domain experts to probe how AR could provide solutions to support their collaborative efforts. We focused on using tangible artifacts in the form of storyboards to create a shared understanding with target users in remote collaboration. Afterwards, we identify a set of requirements, which we materialize through the design and creation of a collaborative prototype based on sharing of enhanced AR annotations. Finally, we present and discuss the results from a case study on a maintenance context, which provides interesting insights that can be applied to other remote settings, thus facilitating the digitization of the industry sector.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Adcock, M., Ranatunga, D., Smith, R., Thomas, B.H.: Object-based touch manipulation for remote guidance of physical tasks. In: ACM Symposium on Spatial User Interaction, pp. 113–122 (2014)

  2. del Amo, I.F., Erkoyuncu, J.A., Roy, R., Palmarini, R., Onoufriou, D.: A systematic review of augmented reality content-related techniques for knowledge transfer in maintenance applications. Comput. Ind. 103, 47–71 (2018)

    Article  Google Scholar 

  3. Ardito, C., Desolda, G., Lanzilotti, R., Malizia, A., Matera, M.: Analysing trade-offs in frameworks for the design of smart environments 39(1), 47–71 (2020)

    Google Scholar 

  4. Aschauer, A., Reisner-Kollmann, I., Wolfartsberger, J.: Creating an open-source augmented reality remote support tool for industry: challenges and learnings. Proc. Comput. Sci. 180, 269–279 (2021)

    Article  Google Scholar 

  5. Ashtari, N., Bunt, A., McGrenere, J., Nebeling, M., Chilana, P.K.: Creating augmented and virtual reality applications: current practices, challenges, and opportunities. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20, pp. 1-13 (2020)

  6. Barnum, C.M.: Usability Testing Essentials: Ready, Set...Test!, 1st edn. Morgan Kaufmann Publishers Inc (2010)

  7. Barroso, J., Fonseca, L., Marques, B., Dias, P., Sousa Santos, B.: Remote collaboration using mixed reality: exploring a shared model approach through different interaction methods, pp. 1–6 (2020)

  8. Belen, R.A.J., Nguyen, H., Filonik, D., Favero, D.D., Bednarz, T.: A systematic review of the current state of collaborative mixed reality technologies: 2013–2018. AIMS Electr. Electr. Eng. 3, 181 (2019)

    Article  Google Scholar 

  9. Bhattacharya, B., Winer, E.H.: Augmented reality via expert demonstration authoring (areda). Comput. Ind. 105, 61–79 (2019)

    Article  Google Scholar 

  10. Billinghurst, M.: Grand challenges for augmented reality 2, 12 (2021)

  11. Bottani, E., Vignali, G.: Augmented reality technology in the manufacturing industry: a review of the last decade. IISE Trans. 51(3), 284–310 (2019)

    Article  Google Scholar 

  12. Bottecchia, S., Cieutat, J.M., Merlo, C., Jessel, J.P.: A new AR interaction paradigm for collaborative teleassistance system: the POA. Int. J. Interact. Des. Manuf. 3(1), 35–40 (2009)

    Article  Google Scholar 

  13. Bruno, F., Barbieri, L., Marino, E., Muzzupappa, M., D’Oriano, L., Colacino, B.: An augmented reality tool to detect and annotate design variations in an Industry 4.0 approach. Int. J. Adv. Manuf. Technol. 105(1), 875–887 (2019)

  14. Ceyssens, J., Coppers, S., Vanacken, D.: Towards traceable design rationale in augmented reality. In: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–8 (2020)

  15. Choi, S.H., Kim, M., Lee, J.Y.: Situation-dependent remote ar collaborations: image-based collaboration using a 3d perspective map and live video-based collaboration with a synchronized vr mode. Comput. Ind. 101, 51–66 (2018)

    Article  Google Scholar 

  16. Courage, C., Baxter, K.: Understanding Your Users: A Practical Guide to User Requirements Methods, Tools, and Techniques (2004)

  17. de Souza Cardoso, L.F., Mariano, F.C.M.Q., Zorzal, E.R.: A survey of industrial augmented reality. Comput. Ind. Eng.139, 106159 (2020)

  18. Dey, A., Billinghurst, M., Lindeman, R.W., Ii, J.E.S.: A systematic review of 10 years of augmented reality usability studies: 2005 to 2014. Front. Robot. AI 5, 1–37 (2018)

    Article  Google Scholar 

  19. Egger, J., Masood, T.: Augmented reality in support of intelligent manufacturing - a systematic literature review. Comput. Ind. Eng. 140,(2020)

  20. Ens, B., Lanir, J., Tang, A., Bateman, S., Lee, G., Piumsomboon, T., Billinghurst, M.: Revisiting collaboration through mixed reality: the evolution of groupware. Int. J. Hum. Comput Stud. 131, 81–98 (2019)

    Article  Google Scholar 

  21. Fakourfar, O., Ta, K., Tang, R., Bateman, S., Tang, A.: Stabilized annotations for mobile remote assistance. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 1548-1560 (2016)

  22. del Amo, I.F., Erkoyuncu, J.A., Roy, R., Wilding, S.: Augmented reality in maintenance: an information-centred design framework. Proc. Manuf. 19, 148–155 (2018)

    Google Scholar 

  23. Ferrari, A., Spoletini, P., Gnesi, S.: Ambiguity and tacit knowledge in requirements elicitation interviews 21(3), 333–355 (2016)

  24. Ferrise, F., Caruso, G., Bordegoni, M.: Multimodal training and tele-assistance systems for the maintenance of industrial products: This paper presents a multimodal and remote training system for improvement of maintenance quality in the case study of washing machine. Virtual Phys. Prototyp. 8, 113–126 (2013)

    Article  Google Scholar 

  25. Fiorentino, M., Uva, A.E., Gattullo, M., Debernardis, S., Monno, G.: Augmented reality on large screen for interactive maintenance instructions. Comput. Ind. 65(2), 270–278 (2014)

    Article  Google Scholar 

  26. Flotyński, J.: Creating explorable extended reality environments with semantic annotations pp. 1–31 (2020)

  27. García-Pereira, I., Gimeno, J., Morillo, P., Casanova-Salas, P.: A taxonomy of augmented reality annotations. In International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - VISIGRAPP2020, vol. 1, pp. 412–419 (2020)

  28. Gattullo, M., Scurati, G.W., Fiorentino, M., Uva, A.E., Ferrise, F., Bordegoni, M.: Towards augmented reality manuals for industry 4.0: a methodology. Robot. Computer-Integ. Manuf. 56, 276–286 (2019)

    Article  Google Scholar 

  29. Gauglitz, S., Nuernberger, B., Turk, M., Höllerer, T.: World-stabilized annotations and virtual scene navigation for remote collaboration. In ACM Symposium on User Interface Software and Technology, UIST’14, pp. 449–459 (2014)

  30. Geng, J., Song, X., Pan, Y., Tang, J., Liu, Y., Zhao, D., Ma, Y.: A systematic design method of adaptive augmented reality work instruction for complex industrial operations. Comput. Ind. 119, 103229 (2020)

    Article  Google Scholar 

  31. Grudin, J., Poltrock, S.: The encyclopedia of human-computer interaction (2013)

  32. Gurevich, P., Lanir, J., Cohen, B.: Design and implementation of teleadvisor: a projection-based augmented reality system for remote collaboration. Comput. Supported Cooper. Work CSCW Int. J. 24, 527–562 (2015)

    Article  Google Scholar 

  33. Hall, M., McMahon, C.A., Bermell-Garcia, P., Johansson, A., Ravindranath, R.: Capturing synchronous collaborative design activities: A state-of-the-art technology review. In Proceedings of International Design Conference, DESIGN 2018, pp. 347–358 (2018)

  34. Hofmann, E., Rüsch, M.: Industry 4.0 and the current status as well as future prospects on logistics. Comput. Ind. 89, 23–34 (2017)

    Article  Google Scholar 

  35. Irlitti, A., Smith, R.T., Itzstein, S.V., Billinghurst, M., Thomas, B.H.: Challenges for asynchronous collaboration in augmented reality. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), pp. 31–35 (2016)

  36. Jacko, J.A.: Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications, Third Edition (2012)

  37. Jalo, H., Pirkkalainen, H., Torro, O., Kärkkäinen, H., Puhto, J., Kankaanpää, T.: How Can Collaborative Augmented Reality Support Operative Work in the Facility Management Industry? In Proceedings of the 10th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2018), pp. 41–51 (2018)

  38. Jerald, J.: The VR Book: Human-Centered Design for Virtual Reality (2015)

  39. Jetter, J., Eimecke, J., Rese, A.: Augmented reality tools for industrial applications: what are potential key performance indicators and who benefits? Comput. Hum. Behav. 87, 18–33 (2018)

    Article  Google Scholar 

  40. Johnson, S., Gibson, M., Mutlu, B.: Handheld or handsfree? remote collaboration via lightweight head-mounted displays and handheld devices. In ACM Conference on Computer Supported Cooperative Work and Social Computing, pp. 1825-1836 (2015)

  41. Kim, K., Billinghurst, M., Bruder, G., Duh, H.B., Welch, G.F.: Revisiting trends in augmented reality research: a review of the 2nd decade of ismar (2008–2017). IEEE Trans. Vis. Comput. Graph. 24, 2947–2962 (2018)

    Article  Google Scholar 

  42. Kim, S., Billinghurst, M., Kim, K.: Multimodal interfaces and communication cues for remote collaboration. J. Multimodal User Interfaces 14(4), 313–319 (2020)

    Article  Google Scholar 

  43. Kim, S., Billinghurst, M., Lee, C., Lee, G.: Using freeze frame and visual notifications in an annotation drawing interface for remote collaboration. Trans. Internet Inf. Syst. 12, 6034–6056 (2018)

    Google Scholar 

  44. Kim, S., Billinghurst, M., Lee, G.: The effect of collaboration styles and view independence on video-mediated remote collaboration. Comput. Support. Cooper. Work 27, 569–607 (2018)

    Article  Google Scholar 

  45. Kim, S., Lee, G., Billinghurst, M., Huang, W.: The combination of visual communication cues in mixed reality remote collaboration. J. Multimodal User Interfaces 14(4), 321–335 (2020)

    Article  Google Scholar 

  46. Kim, S., Lee, G., Huang, W., Kim, H., Woo, W., Billinghurst, M.: Evaluating the combination of visual communication cues for hmd-based mixed reality remote collaboration. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–13 (2019)

  47. Kim, S., Lee, G.A., Ha, S., Sakata, N., Billinghurst, M.: Automatically freezing live video for annotation during remote collaboration. In: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, pp. 1669-1674 (2015)

  48. Kong, X.T.R., Luo, H., Huang, G.Q., Yang, X.: Industrial wearable system: the human-centric empowering technology in Industry 4.0. J. Intell. Manuf. 30(8), 2853–2869 (2019)

    Article  Google Scholar 

  49. Kraus, V., Boden, A., Oppermann, L., Reiners, R.: Current practices, challenges, and design implications for collaborative ar/vr application development. In: 2021 CHI Conference on Human Factors in Computing Systems (2021)

  50. Leutert, F., Schilling, K.: Augmented reality for telemaintenance and -inspection in force-sensitive industrial robot applications. IFAC-PapersOnLine 48(10), 153–158 (2015)

    Article  Google Scholar 

  51. Liao, Y., Deschamps, F., de Freitas Rocha L.E., Ramos, L.F.P.: Past, present and future of industry 4.0 - a systematic literature review and research agenda proposal. Int. J. Prod. Res. 55(12), 3609–3629 (2017)

  52. Limbu, B.H., Jarodzka, H., Klemke, R., Specht, M.: Using sensors and augmented reality to train apprentices using recorded expert performance: a systematic literature review. Educ. Res. Rev. 25, 1–22 (2018)

    Article  Google Scholar 

  53. Longo, F., Nicoletti, L., Padovano, A.: Computers and industrial engineering. Smart operators in industry 4.0: a human-centered approach to enhance operators capabilities and competencies within the new smart factory context. 113, 144–159 (2017)

  54. van Lopik, K., Sinclair, M., Sharpe, R., Conway, P., West, A.: Developing augmented reality capabilities for industry 40 small enterprises: Lessons learnt from a content authoring case study. Computer. Ind. 117 (2020)

  55. Lorenz, M., Knopp, S., Klimant, P.: Industrial augmented reality: Requirements for an augmented reality maintenance worker support system. In 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pp. 151–153 (2018)

  56. Lukosch, S., Billinghurst, M., Alem, L., Kiyokawa, K.: Computer supported cooperative work, CSCW 2015. Collab. Augment. Real. 24, 515–525 (2015)

    Google Scholar 

  57. Madeira, T., Marques, B., Alves, J., Dias, P., Santos, B.S.: Exploring annotations and hand tracking in augmented reality for remote collaboration. In Human Systems Engineering and Design III, pp. 83–89. Springer International Publishing, Berlin (2021)

  58. Marques, B., Barroso, J., Dias, P., Sousa Santos, B.: Exploring MR for Remote Collaboration : Comparing a 3D shared model approach with 2D annotations. In International Conference on Graphics and Interaction, ICGI 2021, pp. 1–4 (2021)

  59. Marques, B., Silva, S., Dias, P., Santos, B.S.: A toolkit to facilitate evaluation and characterization of the collaborative process in scenarios of remote assistance supported by AR. In IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), pp. 336–337 (2021)

  60. Marques, B., Silva, S., Dias, P., Santos, B.S.: An ontology for evaluation of remote collaboration using augmented reality. In: European Conference on Computer-Supported Cooperative Work, ECSCW: The International Venue on Practice-centred Computing on the Design of Cooperation Technologies - Posters and Demos, Reports of the European Society for Socially Embedded Technologies, pp. 1–8 (2021)

  61. Marques, B., Silva, S.S., Alves, J., Araujo, T., Dias, P., Santos, B.S.: A conceptual model and taxonomy for collaborative augmented reality. IEEE Trans. Vis. Comput. Graph. pp. 1–18 (2021)

  62. Marques, B., Teixeira, A., Silva, S., Alves, J., Dias, P., Santos, B.S.: A conceptual model for data collection and analysis for ar-based remote collaboration evaluation. In IEEE International Symposium on Mixed and Augmented Reality, ISMAR (2020)

  63. Marques, B., Teixeira, A., Silva, S., Alves, J., Dias, P., Santos, B.S.: A critical analysis on remote collaboration mediated by augmented reality: making a case for improved characterization and evaluation of the collaborative process. Comput. Graph. pp. 1–17 (2021)

  64. Masoni, R., Ferrise, F., Bordegoni, M., Gattullo, M., Uva, A.E., Fiorentino, M., Carrabba, E.: Procedia Manufacturing. Supporting remote maintenance in industry 4.0 through augmented reality. 11, 1296–1302 (2017)

  65. Masood, T., Egger, J.: Robotics and Computer-Integrated Manufacturing. Augmented reality in support of industry 4.0–implementation challenges and success factors. 58, 181–195 (2019)

  66. Masood, T., Egger, J.: Adopting augmented reality in the age of industrial digitalisation. Comput. Ind. 115, 103112 (2020)

    Article  Google Scholar 

  67. Hernandez-de Menendez, M., Díaz, C.A.E., Morales-Menendez, R.: Engineering education for smart 4.0 technology: a review. Int. J. Interact. Des. Manuf. 14(3), 789–803 (2020)

  68. Hernandez-de Menendez, M., Morales-Menendez, R., Escobar, C.A., McGovern, M.: Competencies for industry 4.0. Int. J. Interact. Des. Manuf. 14(4), 1511–1524 (2020)

    Article  Google Scholar 

  69. Merino, L., Schwarzl, M., Kraus, M., Sedlmair, M., Schmalstieg, D., Weiskopf, D.: Evaluating mixed and augmented reality: a systematic literature review (2009–2019). In IEEE International Symposium on Mixed and Augmented Reality, ISMAR, pp. 438–451 (2020)

  70. Meyer, M., Dykes, J.: Criteria for rigor in visualization design study 26(1), 87–97 (2019)

  71. Miaskiewicz, T., Kozar, K.A.: Personas and user-centered design: How can personas benefit product design processes? Des. Stud. 32(5), 417–430 (2011)

    Article  Google Scholar 

  72. Mourtzis, D., Zogopoulos, V., Vlachou, E.: Augmented reality application to support remote maintenance as a service in the robotics industry. Proc. CIRP 63, 46–51 (2017)

    Article  Google Scholar 

  73. Nielsen, J., Landauer, T.K.: A mathematical model of the finding of usability problems. In Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems, pp. 206–213 (1993)

  74. Nielsen, L.: Personas - User Focused Design. Springer Publishing Company Inc, Berlin (2012)

    Google Scholar 

  75. Oda, O., Elvezio, C., Sukan, M., Feiner, S., Tversky, B.: Virtual replicas for remote assistance in virtual and augmented reality. In ACM Symposium on User Interface Software and Technology, UIST, pp. 405–415 (2015)

  76. Ong, S.K., Yuan, M.L., Nee, A.Y.C.: Augmented reality applications in manufacturing: a survey. Int. J. Prod. Res. 46(10), 2707–2742 (2008)

    Article  MATH  Google Scholar 

  77. Palmarini, R., Erkoyuncu, J.A., Roy, R., Torabmostaedi, H.: A systematic review of augmented reality applications in maintenance. Robot. Computer-Integr. Manuf. 49, 215–228 (2018)

    Article  Google Scholar 

  78. Piumsomboon, T., Dey, A., Ens, B., Lee, G., Billinghurst, M.: The effects of sharing awareness cues in collaborative mixed reality. Front. Robot. AI 6, 1–5 (2019)

  79. Plummer, P.: Focus group methodology. Part 1: design considerations. Int. J. Therapy Rehabil. 24, 297–301 (2017)

    Article  Google Scholar 

  80. Quandt, M., Knoke, B., Gorldt, C., Freitag, M., Thoben, K.D.: General requirements for industrial augmented reality applications. Proc. CIRP 72, 1130–1135 (2018)

    Article  Google Scholar 

  81. Ranatunga, D., Feng, D., Adcock, M., Thomas, B.: Towards object based manipulation in remote guidance. In 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 1–6 (2013)

  82. Röltgen, D., Dumitrescu, R.: Classification of industrial augmented reality use cases. Proc. CIRP 91, 93–100 (2020)

    Article  Google Scholar 

  83. Smith, J.A.: Qualitative psychology: a practical guide to research methods (2015)

  84. Smullin, M.M., Haapala, K.R., Mani, M., Morris, K.C.: Using industry focus groups and literature review to identify challenges in sustainable assessment theory and practice. Int. Des. Eng. Tech. Conf. Comput. Inf. Eng. Conf. 4, 1–10 (2016)

    Google Scholar 

  85. Talkad Sukumar, P., Avellino, I., Remy, C., DeVito, M.A., Dillahunt, T.R., McGrenere, J., Wilson, M.L.: Transparency in qualitative research: increasing fairness in the chi review process. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–6 (2020)

  86. Teo, T., Lawrence, L., Lee, G.A., Billinghurst, M., Adcock, M.: Mixed reality remote collaboration combining 360 video and 3D reconstruction. In ACM CHI Conference on Human Factors in Computing Systems, CHI 2019, pp. 1–14 (2019)

  87. Tullis, T., Albert, W.: Measuring the User Experience, Collecting, Analyzing, and Presenting Usability Metrics. 2nd edn. Morgan Kaufmann Publishers Inc. (2013)

  88. Tzimas, E., Vosniakos, G.C., Matsas, E.: Machine tool setup instructions in the smart factory using augmented reality: a system construction perspective. Int. J. Interact. Des. Manuf. 13(1), 121–136 (2019)

    Article  Google Scholar 

  89. Wang, X., Ong, S.K., Nee, A.Y.C.: A comprehensive survey of augmented reality assembly research. Adv. Manuf. 4(1), 1–22 (2016)

    Article  Google Scholar 

  90. Zhu, J., Ong, S.K., Nee, A.Y.: A context-aware augmented reality system to assist the maintenance operators. Int. J. Interact. Des. Manuf. 8(4), 293–304 (2014)

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank the reviewers for their thoughtful comments and suggestions towards improving on an earlier version of this manuscript. To everyone involved in case studies, and discussion groups, thanks for your time and expertise. The authors would like to acknowledge the support provided by Andreia Santos. This research was developed in the scope of the PhD grant [SFRH/BD/143276/2019], funded by FCT - Foundation for Science and Technology. The maintenance context of this research is based on an industrial collaboration under the Smart Green Homes Project [POCI-01-0247-FEDER-007678], a co-promotion between Bosch Termotecnologia S.A. and the University of Aveiro. It is financed by Portugal 2020 under the Competitiveness and Internationalization Operational Program, and by the European Regional Development Fund. It was also supported by IEETA - Institute of Electronics and Informatics Engineering of Aveiro, funded by National Funds through FCT, in the context of the project [UID/CEC/00127/2019].

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Bernardo Marques.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Marques, B., Silva, S., Alves, J. et al. Remote collaboration in maintenance contexts using augmented reality: insights from a participatory process. Int J Interact Des Manuf 16, 419–438 (2022). https://doi.org/10.1007/s12008-021-00798-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12008-021-00798-6

Keywords

Navigation