Improving maintenance efficiency and safety through a human-centric approach

Advances in Manufacturing volume 9, pages104–114(2021)Cite this article

Abstract

This paper presents an adaptive human-machine interface (HMI) that can provide appropriate sets of digital maintenance information and guidance to an operator during maintenance. It takes into consideration the expertise level of the operator and the maintenance context and progress. The proposed human-centric methodology considers the heart rate, intention, and expertise level of the operator, which can be captured using sensors during maintenance. A set of rules is formulated based on the sensor data to infer the state of the operator during a maintenance task. Based on the operator state, the adaptive HMI can augment the operator’s senses using a scheme that combines visual, audio, and haptic guidance cues during maintenance to enhance the operator’s ability to perceive information and perform maintenance tasks. Various schemes of visual, audio, and haptic cues are developed based on a comparison of the best practices obtained from experienced operators.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

US$ 39.95

Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

References

  1. 1.

    Rødseth H, Schjølberg P, Marhaug A (2017) Deep digital maintenance. Adv Manuf 5:299–310

    Article  Google Scholar 

  2. 2.

    Wang X, Ong SK, Nee AYC (2016) A comprehensive survey of augmented reality assembly research. Adv Manuf 4:1–22

    Article  Google Scholar 

  3. 3.

    Khalek IA, Chalhoub JM, Ayer SK (2019) Augmented reality for identifying maintainability concerns during design. Adv Civil Eng 3:8547928. https://doi.org/10.1155/2019/8547928

    Article  Google Scholar 

  4. 4.

    Gattullo M, Scurati GW, Evangelista A et al (2020) Informing the use of visual assets in industrial augmented reality. In: Proceedings of the international conference on design tools and methods in industrial engineering, 9–10 September, Modena, Italy, pp 106–117

  5. 5.

    Martinettia A, Rajabalinejada M, van Dongena L (2017) Shaping the future maintenance operations: reflections on the adoptions of augmented reality through problems and opportunities. Proc CIRP 59:4–17

    Google Scholar 

  6. 6.

    Abramovici M, Wolf M, Adwernat S et al (2017) Context-aware maintenance support for augmented reality assistance and synchronous multi-user collaboration. Proc CIRP 59:18–22

    Article  Google Scholar 

  7. 7.

    Gattullo M, Dalena V, Evangelista A et al (2019) A context-aware technical information manager for presentation in augmented reality. In: 2019 IEEE conference on virtual reality and 3D user interfaces, 23–27 March, Osaka, Japan, pp 939–940

  8. 8.

    Caudell TP, Mizell DW (1992) Augmented reality: an application of heads-up display technology to manual manufacturing processes. In: Proceedings of the 25th international conference on system sciences, Kauai, Hawaii, 7–10 January, pp 659–669

  9. 9.

    Friedrich W, Jahn D, Schmidt L (2002) ARVIKA: augmented reality for development, production and service. In: The 1st international symposium on mixed and augmented reality (ISMAR), Darmstadt, 30 September 1 October, pp 3–4. https://doi.org/10.1109/ismar.2002.1115059

  10. 10.

    De Groote P (1995) Maintenance performance analysis: a practical approach. J Qual Maint Eng 1(2):4–24

    Article  Google Scholar 

  11. 11.

    Dini G, Mura MD (2015) Application of augmented reality techniques in through-life engineering services. Proc CIRP 38:14–23. https://doi.org/10.1016/j.procir.2015.07.044

    Article  Google Scholar 

  12. 12.

    Regenbrecht HT, Specht R (2000) A mobile passive augmented reality device-Mpard. In: ISAR 2000, Munich, Germany, 5–6 October, pp 81–84. https://doi.org/10.1109/isar.2000.880926

  13. 13.

    Ayala GA, Bobadilla IG, Figueroa GA et al (2016) Virtual reality training system for maintenance and operation of high-voltage overhead power lines. Virtual Reality 20(1):1–14. https://doi.org/10.1007/s10055-015-0280-6

    Article  Google Scholar 

  14. 14.

    Jetter J, Eimecke J, Rese A (2018) Augmented reality tools for industrial applications: what are potential key performance indicators and who benefits? Comput Hum Behav 87:18–33

    Article  Google Scholar 

  15. 15.

    Oliveira R, Farinha T, Raposo H et al (2014) Augmented reality and the future of maintenance. In: Proceedings of maintenance performance measurement and management (MPMM) conference, 4–5 September, Coimbra, pp 81–88

  16. 16.

    Gattullo M, Uva AE, Fiorentino M et al (2015) Legibility in industrial AR: text style, color coding, and illuminance. IEEE Comput Gr Appl 35(2):52–61

    Article  Google Scholar 

  17. 17.

    Dünser A, Grasset R, Seichter H et al (2007) Applying HCI principles to AR systems design. In: Workshop proceedings of mixed reality user interfaces: specification, authoring, adaptation (MRUI’07), 11 Mar, Charlotte, NC, USA, pp 37–42

  18. 18.

    Müller T (2015) Towards a framework for information presentation in augmented reality for the support of procedural tasks. In: International conference on augmented and virtual reality, 31 August 3 September, Lecce, Italy, pp 490–497

  19. 19.

    Aschenbrenner D, Latoschik ME, Schilling K (2016) Industrial maintenance with augmented reality: two case studies. In: Proceedings of the 22nd ACM conference on virtual reality software and technology (VRST16). ACM Press, Garching, Germany, pp 341–342

  20. 20.

    Mourtzisa D, Zogopoulos V, Vlachou E (2017) Augmented reality application to support remote maintenance as a service in the robotics industry. Proc CIRP 63:46–51

    Article  Google Scholar 

  21. 21.

    Kammerer K, Pryss R, Sommer K et al (2018) Towards context-aware process guidance in cyber-physical systems with augmented reality. In: The 4th international workshop on requirements engineering for self-adaptive, collaborative, and cyber physical systems, 20 Auguest, Banff, Canada, pp 44–51

  22. 22.

    Zhu J, Ong SK, Nee AYC (2015) A context-aware augmented reality assisted maintenance system. Int J Comput Integr Manuf 28(2):213–225

    Article  Google Scholar 

  23. 23.

    Erkoyuncu JA, del Amo IF, Dalle MM et al (2017) Improving efficiency of industrial maintenance with context aware adaptive authoring in augmented reality. Ann CIRP 66:465–468

    Article  Google Scholar 

  24. 24.

    Hervás R, Bravo J, Fontecha J et al (2013) Achieving adaptive augmented reality through ontological context-awareness applied to AAL scenarios. J Univ Comput Sci 19(9):1334–1349

    Google Scholar 

  25. 25.

    Holm M, Danielsson O, Syberfeldt A et al (2017) Adaptive instructions to novice shop-floor operators using augmented reality. J Ind Prod Eng 34(5):362–374

    Google Scholar 

  26. 26.

    Syberfeldt A, Danielsson O, Holm M et al (2016) Dynamic operator instructions based on augmented reality and rule-based expert systems. Proc CIRP 41:346–351

    Article  Google Scholar 

  27. 27.

    Lages WS, Bowman DA (2019) Walking with adaptive augmented reality workspaces: design and usage patterns. In: The 24th international conference on intelligent user interfaces, 17–20 March, CM, New York, USA, pp 356–366

  28. 28.

    Petrangeli S, Simon G, Wang H et al (2019) Dynamic adaptive streaming for augmented reality applications. In: IEEE international symposium on multimedia (ISM), 9–11 December, San Diego, California, USA, pp 56–63

  29. 29.

    Tsai CH, Huang JY (2018) Augmented reality display based on user behavior. Comput Stand Interfaces 55:171–181

    Article  Google Scholar 

  30. 30.

    Healey JA, Picard RW (2005) Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans Intelligent Transp Syst 6(2):156–166

    Article  Google Scholar 

  31. 31.

    Knopfle C, Weidenhausen J, Chauvigne L et al (2005) Template based authoring for AR based service scenarios. In: IEEE proceedings of virtual reality, 12–16 March, Bonn, Germany

  32. 32.

    Gattullo M, Uva AE, Fiorentino M et al (2017) From paper manual to AR manual: do we still need text? Proc Manuf 11:1303–1310

    Google Scholar 

  33. 33.

    Masoni R, Ferrise F, Bordegoni M et al (2017) Supporting remote maintenance in Industry 4.0 through augmented reality. Proc Manuf 11:1296–1302

    Google Scholar 

  34. 34.

    Jo GS, Oh KJ, Ha I et al (2014) A unified framework for augmented reality and knowledge-based systems in maintaining aircraft. In: Proceedings of the national conference on artificial intelligence, 27–31 July, Quebec, Canada, pp 2990–2997

  35. 35.

    Palmarini R, Erkoyuncu JA, Roy R et al (2018) A systematic review of augmented reality applications in maintenance. Robot Comput Integr Manuf 49:215–228

    Article  Google Scholar 

Download references

Author information

Affiliations

  1. Graduate School for Integrative Sciences and Engineering, National University of Singapore, 28 Medical Drive, Singapore, 117456, Singapore

    C. Y. Siew, S. K. Ong & A. Y. C. Nee

  2. Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore, 117576, Singapore

    S. K. Ong & A. Y. C. Nee

Authors
  1. C. Y. Siew
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. K. Ong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. A. Y. C. Nee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to S. K. Ong.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Siew, C.Y., Ong, S.K. & Nee, A.Y.C. Improving maintenance efficiency and safety through a human-centric approach. Adv. Manuf. 9, 104–114 (2021). https://doi.org/10.1007/s40436-020-00334-x

Download citation

Keywords

  • Augmented reality (AR)
  • Ergonomics
  • Product development
  • Maintenance