Machine tool setup instructions in the smart factory using augmented reality: a system construction perspective

  • Evangelos Tzimas
  • George-Christopher Vosniakos
  • Elias Matsas
Original Paper


The concept of smart factory includes collection and distribution of information in real time to help human operators in their work. Machine setup instructions are notable examples of such information. This paper advocates the use of augmented reality (AR) to present such information addressing in an intuitive manner the demonstration of setup tasks on the variety of machine tools that are available in a factory and for the multitude of parts that are produced. Turning and machining centres are used as typical machine tools, for which setup instruction applications were created on a widely used computer game development platform enhanced by standard AR functionality based on target markers present on the scene. This AR demonstrator has been primarily created as a proof of concept that highlights the feasibility of such low-cost, efficient and user friendly AR applications of industrial guidance and training. The emphasis of the work is on technical aspects of AR application development, which is lacking in literature. In particular, interfacing of distance sensors that impart intelligence, scenario structuring in terms of states and state threads, as well as techniques ensuring expansibility, flexibility and ease of authoring of such applications are demonstrated.


Augmented reality Manufacturing execution Machine tool Setup Smart factory Industry 4.0 


  1. 1.
    Billinghurst, M., Clark, A., Lee, G.: A survey of augmented reality. Found. Trends® Hum.–Comput. Interact. 8(2–3), 73–272 (2015)CrossRefGoogle Scholar
  2. 2.
    van Krevelen, D.W.F., Poelman, R.: A survey of augmented reality technologies, applications and limitations. Int. J. Virtual Real. 9(2), 1–20 (2010)Google Scholar
  3. 3.
    Fite-Georgel, P.: Is there a reality in industrial augmented reality? In: 2011 10th IEEE International Symposium on Mixed and Augmented Reality, pp. 201–210 (2011)Google Scholar
  4. 4.
    Elia, V., Gnoni, M.G., Lanzilotto, A.: Evaluating the application of augmented reality devices in manufacturing from a process point of view: an AHP based model. Expert Syst. Appl. 63, 187–197 (2016)CrossRefGoogle Scholar
  5. 5.
    Davies, P., Lee, D.: Augmented Reality in Manufacturing at the Boeing Company Lessons Learned and Future Directions (2014)Google Scholar
  6. 6.
    Yew, A.W.W., Ong, S.K., Nee, A.Y.C.: Towards a griddable distributed manufacturing system with augmented reality interfaces. Robot. Comput. Integr. Manuf. 39, 43–55 (2016)CrossRefGoogle Scholar
  7. 7.
    Gorecky, D., Mura, K., Arlt, F.: A vision on training and knowledge sharing applications in future factories. IFAC Proc. 46(15), 90–97 (2013)CrossRefGoogle Scholar
  8. 8.
    Doshi, A., Smith, R.T., Thomas, B.H., Bouras, C.: Use of projector based augmented reality to improve manual spot-welding precision and accuracy for automotive manufacturing. Int. J. Adv. Manuf. Technol. 89(5–8), 1279–1293 (2016)Google Scholar
  9. 9.
    Syberfeldt, A., Danielsson, O., Holm, M., Wang, L.: Visual assembling guidance using augmented reality. Procedia Manuf. 1, 98–109 (2015)CrossRefGoogle Scholar
  10. 10.
    Fiorentino, M., Uva, A.E., Gattullo, M., Debernardis, S., Monno, G.: Augmented reality on large screen for interactive maintenance instructions. Comput. Ind. 65(2), 270–278 (2014)CrossRefGoogle Scholar
  11. 11.
    Alvarez, H., Aguinaga, I., Borro, D.: Providing guidance for maintenance operations using automatic markerless Augmented Reality system. In: 2011 10th IEEE International Symposium on Mixed and Augmented Reality, pp. 181–190 (2011)Google Scholar
  12. 12.
    Wang, X., Ong, S.K., Nee, A.Y.C.: A comprehensive survey of augmented reality assembly research. Adv. Manuf. 4(1), 1–22 (2016)CrossRefGoogle Scholar
  13. 13.
    Radkowski, R., Herrema, J., Oliver, J.: Augmented reality-based manual assembly support with visual features for different degrees of difficulty. Int. J. Hum. Comput. Interact. 31(5), 337–349 (2015)CrossRefGoogle Scholar
  14. 14.
    Lamberti, F., Manuri, F., Sanna, A., Paravati, G., Pezzolla, P., Montuschi, P.: Challenges, opportunities, and future trends of emerging techniques for augmented reality-based maintenance. IEEE Trans. Emerg. Top. Comput. 2(4), 411–421 (2014)CrossRefGoogle Scholar
  15. 15.
    Pirvu, B.-C., Zamfirescu, C.-B., Gorecky, D.: Engineering insights from an anthropocentric cyber-physical system: a case study for an assembly station. Mechatronics 34, 147–159 (2016)CrossRefGoogle Scholar
  16. 16.
    Rentzos, L., Papanastasiou, S., Papakostas, N., Chryssolouris, G.: Augmented reality for human-based assembly: using product and process semantics. IFAC Proc. 46(15), 98–101 (2013)CrossRefGoogle Scholar
  17. 17.
    Syberfeldt, A., Danielsson, O., Holm, M., Wang, L.: Dynamic operator instructions based on augmented reality and rule-based expert systems. Procedia CIRP 41, 346–351 (2016)CrossRefGoogle Scholar
  18. 18.
    Jo, G.S., Oh, K.J., Ha, I., Lee, K.S., Hong, M.D., Neumann, U., You, S.: A unified framework for augmented reality and knowledge-based systems in maintaining aircraft. Proc. Natl. Conf. Artif. Intell. 4(October 2015), 2990–2997 (2014)Google Scholar
  19. 19.
    Flatt, H., Koch, N., Rocker, C., Gunter, A., Jasperneite, J.: A context-aware assistance system for maintenance applications in smart factories based on augmented reality and indoor localization. In: 2015 IEEE 20th Conference on Emerging Technologies & Factory Automation (ETFA), pp. 1–4 (2015)Google Scholar
  20. 20.
    Gorecky, D., Schmitt, M., Loskyll, M., Zühlke, D.: Human–machine-interaction in the industry 4.0 era. In: Proceedings - 2014 12th IEEE International Conference on Industrial Informatics (INDIN 2014), pp. 289–294 (2014)Google Scholar
  21. 21.
    Rodriguez, L., Quint, F., Gorecky, D., Romero, D., Siller, H.R.: Developing a mixed reality assistance system based on projection mapping technology for manual operations at assembly workstations. Procedia Comput. Sci. 75, 327–333 (2015)CrossRefGoogle Scholar
  22. 22.
    Mura, M.D., Dini, G., Failli, F.: An integrated environment based on augmented reality and sensing device for manual assembly workstations. Procedia CIRP 41, 340–345 (2016)CrossRefGoogle Scholar
  23. 23.
    Kollatsch, C., Schumann, M., Klimant, P., Wittstock, V., Putz, M.: Mobile augmented reality based monitoring of assembly lines. Procedia CIRP 23(C), 246–251 (2014)CrossRefGoogle Scholar
  24. 24.
    Zhou, J., Lee, I., Thomas, B., Menassa, R., Farrant, A., Sansome, A.: Applying spatial augmented reality to facilitate in-situ support for automotive spot welding inspection. In: Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry—(VRCAI’11), p. 195 (2011)Google Scholar
  25. 25.
    ART: ART implements ‘Window To The World’ at AUDIGoogle Scholar
  26. 26.
    Wang, J., Feng, Y., Zeng, C., Li, S.: An augmented reality based system for remote collaborative maintenance instruction of complex products. In: 2014 IEEE International Conference on Automation Science and Engineering (CASE), pp. 309–314 (2014)Google Scholar
  27. 27.
    Garza, L.E., Pantoja, G., Ramírez, P., Ramírez, H., Rodríguez, N., González, E., Quintal, R., Pérez, J.A.: Augmented reality application for the maintenance of a flapper valve of a fuller-kynion type m pump. Procedia Comput. Sci. 25, 154–160 (2013)CrossRefGoogle Scholar
  28. 28.
    Rolim, C., Schmalstieg, D., Kalkofen, D., Teichrieb, V.: Design guidelines for generating augmented reality instructions. In: 2015 IEEE International Symposium on Mixed and Augmented Reality, pp. 120–123 (2015)Google Scholar
  29. 29.
    Petersen, N., Stricker, D.: Cognitive augmented reality. Comput. Graph. 53, 82–91 (2015)CrossRefGoogle Scholar
  30. 30.
    Ramirez, H., Mendivil, E.G., Flores, P.R., Gonzalez, M.C.: Authoring software for augmented reality applications for the use of maintenance and training process. Procedia Comput. Sci. 25, 189–193 (2013)CrossRefGoogle Scholar
  31. 31.
    Paelke, V.: Augmented reality in the smart factory: supporting workers in an industry 4.0. environment. In: Proceedings of the 2014 IEEE Emerging Technology and Factory Automation (ETFA), pp. 1–4 (2014)Google Scholar
  32. 32.
    Loch, F., Quint, F., Brishtel, I.: Comparing video and augmented reality assistance in manual assembly. In: 2016 12th International Conference on Intelligent Environments (IE), pp. 147–150 (2016)Google Scholar
  33. 33.
    Webel, S., Bockholt, U., Engelke, T., Gavish, N., Olbrich, M., Preusche, C.: An augmented reality training platform for assembly and maintenance skills. Robot. Auton. Syst. 61(4), 398–403 (2013)CrossRefGoogle Scholar
  34. 34.
    Monostori, L.: Cyber-physical production systems: roots, expectations and R&D challenges. Procedia CIRP 17, 9–13 (2014)CrossRefGoogle Scholar
  35. 35.
    PTC: Unity API ReferenceGoogle Scholar
  36. 36.
    Maskrey, M., Wang, W.: Interacting with augmented reality. In: Pro iPhone Development with Swift 4, pp. 419–446. Apress, Berkeley, CA (2018)Google Scholar

Copyright information

© Springer-Verlag France SAS, part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of Mechanical EngineeringNational Technical University of AthensAthensGreece

Personalised recommendations