Abstract
This paper presents an adaptive human-machine interface (HMI) that can provide appropriate sets of digital maintenance information and guidance to an operator during maintenance. It takes into consideration the expertise level of the operator and the maintenance context and progress. The proposed human-centric methodology considers the heart rate, intention, and expertise level of the operator, which can be captured using sensors during maintenance. A set of rules is formulated based on the sensor data to infer the state of the operator during a maintenance task. Based on the operator state, the adaptive HMI can augment the operator’s senses using a scheme that combines visual, audio, and haptic guidance cues during maintenance to enhance the operator’s ability to perceive information and perform maintenance tasks. Various schemes of visual, audio, and haptic cues are developed based on a comparison of the best practices obtained from experienced operators.
Similar content being viewed by others
References
Rødseth H, Schjølberg P, Marhaug A (2017) Deep digital maintenance. Adv Manuf 5:299–310
Wang X, Ong SK, Nee AYC (2016) A comprehensive survey of augmented reality assembly research. Adv Manuf 4:1–22
Khalek IA, Chalhoub JM, Ayer SK (2019) Augmented reality for identifying maintainability concerns during design. Adv Civil Eng 3:8547928. https://doi.org/10.1155/2019/8547928
Gattullo M, Scurati GW, Evangelista A et al (2020) Informing the use of visual assets in industrial augmented reality. In: Proceedings of the international conference on design tools and methods in industrial engineering, 9–10 September, Modena, Italy, pp 106–117
Martinettia A, Rajabalinejada M, van Dongena L (2017) Shaping the future maintenance operations: reflections on the adoptions of augmented reality through problems and opportunities. Proc CIRP 59:4–17
Abramovici M, Wolf M, Adwernat S et al (2017) Context-aware maintenance support for augmented reality assistance and synchronous multi-user collaboration. Proc CIRP 59:18–22
Gattullo M, Dalena V, Evangelista A et al (2019) A context-aware technical information manager for presentation in augmented reality. In: 2019 IEEE conference on virtual reality and 3D user interfaces, 23–27 March, Osaka, Japan, pp 939–940
Caudell TP, Mizell DW (1992) Augmented reality: an application of heads-up display technology to manual manufacturing processes. In: Proceedings of the 25th international conference on system sciences, Kauai, Hawaii, 7–10 January, pp 659–669
Friedrich W, Jahn D, Schmidt L (2002) ARVIKA: augmented reality for development, production and service. In: The 1st international symposium on mixed and augmented reality (ISMAR), Darmstadt, 30 September 1 October, pp 3–4. https://doi.org/10.1109/ismar.2002.1115059
De Groote P (1995) Maintenance performance analysis: a practical approach. J Qual Maint Eng 1(2):4–24
Dini G, Mura MD (2015) Application of augmented reality techniques in through-life engineering services. Proc CIRP 38:14–23. https://doi.org/10.1016/j.procir.2015.07.044
Regenbrecht HT, Specht R (2000) A mobile passive augmented reality device-Mpard. In: ISAR 2000, Munich, Germany, 5–6 October, pp 81–84. https://doi.org/10.1109/isar.2000.880926
Ayala GA, Bobadilla IG, Figueroa GA et al (2016) Virtual reality training system for maintenance and operation of high-voltage overhead power lines. Virtual Reality 20(1):1–14. https://doi.org/10.1007/s10055-015-0280-6
Jetter J, Eimecke J, Rese A (2018) Augmented reality tools for industrial applications: what are potential key performance indicators and who benefits? Comput Hum Behav 87:18–33
Oliveira R, Farinha T, Raposo H et al (2014) Augmented reality and the future of maintenance. In: Proceedings of maintenance performance measurement and management (MPMM) conference, 4–5 September, Coimbra, pp 81–88
Gattullo M, Uva AE, Fiorentino M et al (2015) Legibility in industrial AR: text style, color coding, and illuminance. IEEE Comput Gr Appl 35(2):52–61
Dünser A, Grasset R, Seichter H et al (2007) Applying HCI principles to AR systems design. In: Workshop proceedings of mixed reality user interfaces: specification, authoring, adaptation (MRUI’07), 11 Mar, Charlotte, NC, USA, pp 37–42
Müller T (2015) Towards a framework for information presentation in augmented reality for the support of procedural tasks. In: International conference on augmented and virtual reality, 31 August 3 September, Lecce, Italy, pp 490–497
Aschenbrenner D, Latoschik ME, Schilling K (2016) Industrial maintenance with augmented reality: two case studies. In: Proceedings of the 22nd ACM conference on virtual reality software and technology (VRST16). ACM Press, Garching, Germany, pp 341–342
Mourtzisa D, Zogopoulos V, Vlachou E (2017) Augmented reality application to support remote maintenance as a service in the robotics industry. Proc CIRP 63:46–51
Kammerer K, Pryss R, Sommer K et al (2018) Towards context-aware process guidance in cyber-physical systems with augmented reality. In: The 4th international workshop on requirements engineering for self-adaptive, collaborative, and cyber physical systems, 20 Auguest, Banff, Canada, pp 44–51
Zhu J, Ong SK, Nee AYC (2015) A context-aware augmented reality assisted maintenance system. Int J Comput Integr Manuf 28(2):213–225
Erkoyuncu JA, del Amo IF, Dalle MM et al (2017) Improving efficiency of industrial maintenance with context aware adaptive authoring in augmented reality. Ann CIRP 66:465–468
Hervás R, Bravo J, Fontecha J et al (2013) Achieving adaptive augmented reality through ontological context-awareness applied to AAL scenarios. J Univ Comput Sci 19(9):1334–1349
Holm M, Danielsson O, Syberfeldt A et al (2017) Adaptive instructions to novice shop-floor operators using augmented reality. J Ind Prod Eng 34(5):362–374
Syberfeldt A, Danielsson O, Holm M et al (2016) Dynamic operator instructions based on augmented reality and rule-based expert systems. Proc CIRP 41:346–351
Lages WS, Bowman DA (2019) Walking with adaptive augmented reality workspaces: design and usage patterns. In: The 24th international conference on intelligent user interfaces, 17–20 March, CM, New York, USA, pp 356–366
Petrangeli S, Simon G, Wang H et al (2019) Dynamic adaptive streaming for augmented reality applications. In: IEEE international symposium on multimedia (ISM), 9–11 December, San Diego, California, USA, pp 56–63
Tsai CH, Huang JY (2018) Augmented reality display based on user behavior. Comput Stand Interfaces 55:171–181
Healey JA, Picard RW (2005) Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans Intelligent Transp Syst 6(2):156–166
Knopfle C, Weidenhausen J, Chauvigne L et al (2005) Template based authoring for AR based service scenarios. In: IEEE proceedings of virtual reality, 12–16 March, Bonn, Germany
Gattullo M, Uva AE, Fiorentino M et al (2017) From paper manual to AR manual: do we still need text? Proc Manuf 11:1303–1310
Masoni R, Ferrise F, Bordegoni M et al (2017) Supporting remote maintenance in Industry 4.0 through augmented reality. Proc Manuf 11:1296–1302
Jo GS, Oh KJ, Ha I et al (2014) A unified framework for augmented reality and knowledge-based systems in maintaining aircraft. In: Proceedings of the national conference on artificial intelligence, 27–31 July, Quebec, Canada, pp 2990–2997
Palmarini R, Erkoyuncu JA, Roy R et al (2018) A systematic review of augmented reality applications in maintenance. Robot Comput Integr Manuf 49:215–228
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Siew, C.Y., Ong, S.K. & Nee, A.Y.C. Improving maintenance efficiency and safety through a human-centric approach. Adv. Manuf. 9, 104–114 (2021). https://doi.org/10.1007/s40436-020-00334-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40436-020-00334-x