A context-aware augmented reality system to assist the maintenance operators

  • J. Zhu
  • S. K. OngEmail author
  • A. Y. C. Nee
Original Paper


The increasing complexity and the technology advancement of equipment pose challenges to the maintenance operators nowadays. In this paper, a context-aware augmented reality (AR) system is proposed to assist the operators in routine and ad hoc maintenance tasks by providing them with context relevant information. This system integrates AR technology and context-awareness. It is able to analyze the contexts of the maintenance tasks and provide relevant and useful information to the operators through rendering the information on the real equipment and environment. In addition, the authoring of context-aware AR contents, which is essential for developing such applications, is explored. The properties of AR contents and their context-adaptation schemes are discussed, and a low-level authoring approach is proposed for constructing context-aware AR information. Finally, a case study is conducted to evaluate the system performance, and the results demonstrated that the system can adapt to dynamic maintenance situations to provide context relevant information to assist the maintenance operators.


Augmented reality Authoring  Context-awareness  Maintenance and repair 


  1. 1.
    Azuma, R.: A survey of augmented reality. Presence Teleoperators Virtual Environ. 6(4), 355–385 (1997)Google Scholar
  2. 2.
    Bottecchia, S., Cieutat, J.M., Merlo, C., Jessel, J.P.: A new AR interaction paradigm for collaborative teleassistance system: the POA. Int. J. Interact. Design Manuf. 3(1), 35–40 (2009)CrossRefGoogle Scholar
  3. 3.
    Morkos, B., Taiber, J., Summers, J., Mears, L., Fadel, G., Rilka, T.: Mobile devices within manufacturing environments: a BMW applicability study. Int. J. Interact. Design Manuf. 6(2), 101–111 (2012)CrossRefGoogle Scholar
  4. 4.
    Dey, A.K.: Understanding and using context. Pers. Ubiquitous Comput. 5(1), 4–7 (2001)CrossRefGoogle Scholar
  5. 5.
    Byun, H.E., Cheverst, K.: Utilizing context history to provide dynamic adaptations. Appl. Artif. Intell. 18(6), 533–548 (2004)CrossRefGoogle Scholar
  6. 6.
    Zhang, J.X., Sheng, Y.H., Hao, W., Wang, P.P., Tian, P., Miao, K., Pickering, C.K.: A context-aware framework supporting complex ubiquitous scenarios with augmented reality enabled. In: Proceedings of 5th International Conference on Pervasive Computing and Applications, Maribor, pp. 69–74 (2010)Google Scholar
  7. 7.
    Lewandowski, J., Arochena, H.E., Naguib, R.N.G., Chao, K.M.: A portable framework design to support user context aware augmented reality applications. In: Proceedings of 3rd International Conference on Games and Virtual Worlds for Serious Applications, Athens, pp. 144–147 (2011)Google Scholar
  8. 8.
    Shin, C., Kim, H., Kang, C., Jang, Y., Choi, A., Woo, W.: Unified context-aware augmented reality application framework for user-driven tour guides. In: Proceedings of 2010 International Symposium on Ubiquitous Virtual Reality, Gwangju, pp. 52–55 (2010)Google Scholar
  9. 9.
    Feiner, S., Macintyre, B., Seligmann, D.: Knowledge-based augmented reality. Commun. ACM 36, 53–62 (1993)CrossRefGoogle Scholar
  10. 10.
    Friedrich, W.: ARVIKA-augmented reality for development. Production and Service, International Symposium on Mixed and Augmented Reality, Darmstadt, , pp. 3–4 (2002)Google Scholar
  11. 11.
    Schwald, B., et al.: STARMATE: using augmented reality technology for computer guided maintenance of complex mechanical elements. In: Proceedings of eBusiness and eWork Conference (e2001), Venice, pp 17–19 (2001)Google Scholar
  12. 12.
    Savioja, P., Järvinen, P., Karhela, T., Siltanen, P., Woodward, C.: Developing a mobile, service-based augmented reality tool for modern maintenance work. In: Proceedings of the 2nd International Conference on Virtual Reality, Beijing, pp. 554–563 (2007)Google Scholar
  13. 13.
    ARTESAS: Advanced augmented reality technologies for industrial service applications. (2012)
  14. 14.
    Harmo, P., Halme, A., Virekoski, P., Halinen, M., Pitkänen, H.: Etälä—virtual reality assisted telepresence system for remote maintenance. In: Proceedings of the 1st IFAC Conference on Mechatronic Systems, Darmstadt, pp. 1075–1080 (2000)Google Scholar
  15. 15.
    Didier, J.Y., Roussel, D.: AMRA: augmented reality assistance in train maintenance tasks. In: ISMAR’05: Workshop on Industrial Augmented Reality, pp. xvii–xviii (2005)Google Scholar
  16. 16.
    Neubert, J., Pretlove, J., Drummond, T.: Rapidly constructed appearance models for tracking in augmented reality applications. Mach. Vis. Appl. 23(5), 843–856 (2012)CrossRefGoogle Scholar
  17. 17.
    Alvarez, H., Aguinaga, I., Borro, D.: Providing guidance for maintenance operations using automatic markerless augmented reality system. IEEE International Symposium on Mixed and Augmented Reality, Basel, pp. 181–90 (2011)Google Scholar
  18. 18.
    Caponio, A., Hincapie, M., Gonzalez Mendivil, E.: lMAR: highly parallel architecture for markerless augmented reality in aircraft maintenance. International Conference on Virtual and Mixed Reality, Orlando, pp. 20–29 (2011)Google Scholar
  19. 19.
    Yan, W., Ishii, H., Shimoda, H., Izumi, M.: A feasible tracking method of augmented reality for supporting fieldwork of nuclear power plant, Lecture Notes in Computer Science, vol. 5622. LNCS, pp. 639–646 (2009)Google Scholar
  20. 20.
    Klein, G., Murray, D.: Parallel tracking and mapping for small ar workspaces. International Symposium on Mixed and Augmented Reality, Nara (2007)Google Scholar
  21. 21.
    Goose, S., Sudarsky, S., Zhang, X., Navab, N.: Speech-enabled augmented reality supporting mobile industrial maintenance. IEEE Pervasive Comput. 2(1), 65–70 (2003)CrossRefGoogle Scholar
  22. 22.
    Park, H.M., Lee, S.H., Choi, J.S.: Wearable augmented reality system using gaze interaction. 7th IEEE International Symposium on Mixed and Augmented Reality, Cambridge, pp. 175–176 (2008)Google Scholar
  23. 23.
    Yuan, M.L., Ong, S.K., Nee, A.Y.C.: The Virtual Interaction Panel: an easy control tool in augmented reality systems. Comput. Anim. Virtual Worlds J. (Special Issue: The Very Best Papers from CASA 2004 15(3–4):425–432 (2004)Google Scholar
  24. 24.
    Henderson, S.J., Feiner, S.: Opportunistic tangible user interfaces for augmented reality. IEEE Trans. Vis. Comput. Graphics 16(1), 4–16 (2010)CrossRefGoogle Scholar
  25. 25.
    Jacob, R.J.K.: Eye-movement-based human-computer interaction techniques: toward non-command interface, chap. 6. Advances in Human–Computer Interaction, vol. 4. Ablex Publishing Corporation, Norwood, pp. 151–190 (1993)Google Scholar
  26. 26.
    Sakata, N., Kurata, T., Kuzuoka, H.: Visual assist with a laser pointer and wearable display for remote collaboration. Trans. Virtual Real. Soc. Japan 11(4), 561–568 (2006)Google Scholar
  27. 27.
    Ong, S.K., Yuan, M.L., Nee, A.Y.C.: Augmented reality applications in manufacturing: a survey. Int. J. Prod. Res. 46(10), 2707–2742 (2008)CrossRefzbMATHGoogle Scholar
  28. 28.
    HITLabNZ ARToolkit: (2005)
  29. 29.
    Wang, X.H., Zhang, D., Gu, T., Pung, H.H.: Ontology-based context modeling and reasoning using OWL. 2nd IEEE Annual conference on Pervasive Computing and Communications Workshops, Singapore, pp. 18–22 (2004)Google Scholar
  30. 30.
    SWRL: A semantic web rule language combining OWL and RuleML. (retrieved 28 April 2012)
  31. 31.
    Pellet: OWL 2 Reasoner for Java. (retrieved 28 April 2012)
  32. 32.
    Shen, Y., Ong, S.K., Nee, A.Y.C.: Vision-based hand interaction in augmented reality environment. Int. J. Hum. Comput. Interact. 27(6), 523–544 (2011)CrossRefGoogle Scholar
  33. 33.
    The OWL API. (retrieved 28 April 2012)

Copyright information

© Springer-Verlag France 2013

Authors and Affiliations

  1. 1.NUS Graduate School for Integrative Sciences and EngineeringNational University of SingaporeSingapore Singapore
  2. 2.Mechanical Engineering DepartmentNational University of SingaporeSingapore Singapore

Personalised recommendations