Augmented Reality and Natural User Interface Applications for Remote Laboratories

  • Ananda Maiti
  • Mark Smith
  • Andrew D. Maxwell
  • Alexander A. KistEmail author


Augmented reality (AR) has a great potential in creating rich user interfaces where users can view and interact with virtual objects. AR can have both passive objects and active objects. The former do not respond to interaction; the latter can be altered in their orientation, shape and position in relation to other virtual objects, for example. Remote laboratories (RLs) enable access to equipment and experiments via the Internet. This chapter focuses on the use of virtual objects in remote laboratories as an alternative, immersive user interface. Having an AR environment allows users to interact with experiments as virtual objects enabling hands-on experiences. This is made possible by using specialised natural user interface (NUI) devices. These devices can capture the natural movement of users and apply them to virtual objects. This chapter considers the role of AR and NUIs in the context of remote laboratories. It provides the context for AR and NUIs and discusses examples of systems that can be used. It demonstrates how users can interact with remote laboratories in these environments. NUI is part of the curriculum at the undergraduate level. This cyber-physical environment provides an ideal context to teach human-computer interaction (HCI). The first two sections of this chapter describe AR and using NUI in the RL environments. The last section introduces a practical example of using NUI and RL to teach HCI.


Augmented reality E-learning Remote laboratories Computer vision Virtual reality Human-computer interface Kinect 3D sensors Gesture recognition Computer networks 


  1. Andujar, J. M., Mejias, A., & Marquez, M. A. (2011). Augmented reality for the improvement of remote laboratories: An augmented remote laboratory. IEEE Transactions on Education, 54(3), 492–500. Scholar
  2. Apostolellis, P., Bortz, B., Peng, Mi., Polys, N., & Hoegh, A. (2014). Poster: Exploring the integrality and separability of the Leap Motion Controller for direct manipulation 3D interaction. Paper presented at the 3D User Interfaces (3DUI), 2014 IEEE Symposium on.Google Scholar
  3. Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355–385.CrossRefGoogle Scholar
  4. Barfield, W. (2015). Fundamentals of wearable computers and augmented reality. Boca Raton: CRC Press.Google Scholar
  5. Bruder, G., Steinicke, F., & Hinrichs, K. H. (2009). Arch-explore: A natural user interface for immersive architectural walkthroughs. Paper presented at the 3D User Interfaces, 2009. 3DUI 2009. IEEE Symposium on.Google Scholar
  6. Callaghan, M. J., McCusker, K., Losada, J. L., Harkin, J., & Wilson, S. (2013). Using game-based learning in virtual worlds to teach electronic and electrical engineering. IEEE Transactions on Industrial Informatics, 9(1), 575–584. Scholar
  7. Callaghan, M., Savin, M., McShane, N., & Eguiluz, A. (2015). Mapping learning and game mechanics for serious games analysis in engineering education. IEEE Transactions on Emerging Topics in Computing, 99, 1–7.Google Scholar
  8. Chang, Y., Aziz, El-S., Zhang, Z., Zhang, M., Esche, S., & Chassapis, C. (2014). A platform for mechanical assembly education using the Microsoft Kinect. Paper presented at the ASME 2014 International Mechanical Engineering Congress and Exposition.Google Scholar
  9. Corter, J. E., Nickerson, J. V., Esche, S. K., & Chassapis, C. (2004). Remote versus hands-on labs: A comparative study. Paper presented at the Frontiers in Education, 2004. FIE 2004. 34th Annual.Google Scholar
  10. Cubillo, J., Martin, S., Castro, M., & Meier, R. (2012). Control of a remote laboratory by augmented reality. Paper presented at the Teaching, Assessment and Learning for Engineering (TALE), 2012 IEEE International Conference on.Google Scholar
  11. Ester, M., Kriegel, H.-P., Sander, J., & Xu, X. (1996). A density-based algorithm for discovering clusters in large spatial databases with noise. Paper presented at the Second Annual Conference on Knowledge Discovery and Data Mining (KDD-96).Google Scholar
  12. Jara, C. A., Candelas, F. A., Puente, S. T., & Torres, F. (2011). Hands-on experiences of undergraduate students in automatics and robotics using a virtual and remote laboratory. Computers & Education, 57(4), 2451–2461.CrossRefGoogle Scholar
  13. Liu, W. (2010). Natural user interface-next mainstream product user interface. Paper presented at the 2010 IEEE 11th International Conference on Computer-Aided Industrial Design & Conceptual Design 1.Google Scholar
  14. Maier, P., & Klinker, G. (2013). Augmented chemical reactions: 3D interaction methods for chemistry. International Journal of Online Engineering (iJOE), 9(S8), 80. Scholar
  15. Maiti, A., Maxwell, A. D., & Kist, A. A. (2014). Features, trends and characteristics of remote access laboratory management systems. International Journal of Online Engineering, 10(2), 31–37.Google Scholar
  16. Maiti, A., Maxwell, A. D., Kist, A. A., & Orwin, L. (2015). Joining the game and the experiment in peer-to-peer remote laboratories for STEM education. Paper presented at the 2015 3rd Experiment@ International Conference (’15), University of the Azores, Ponta Delgada, Portugal.
  17. Mejías Borrero, A., & Andújar Márquez, J. M. (2012). A pilot study of the effectiveness of augmented reality to enhance the use of remote labs in electrical engineering education. Journal of Science Education and Technology, 21(5), 540–557. Scholar
  18. Melkonyan, A., Gampe, A., Pontual, M., Huang, G., & Akopian, D. (2014). Facilitating remote laboratory deployments using a relay gateway server architecture. IEEE Transactions on Industrial Electronics, 61(1), 477–485. Scholar
  19. Menezes, P., Chouzal, F., Urbano, D., & Restivo, T. (2017). Augmented reality in engineering. In M. E. Auer, D. Guralnick, & J. Uhomoibhi (Eds.), Interactive collaborative learning: Proceedings of the 19th ICL conference – Volume 2 (pp. 221–228). Cham: Springer International Publishing.CrossRefGoogle Scholar
  20. Moore, P., Kist, A. A., Maiti, A., & Maxwell, A. D. (2016). Work in progress: Remote experiment control through gesture recognition. Paper presented at the 2016 13th International Conference on Remote Engineering and Virtual Instrumentation (REV).Google Scholar
  21. Müller, D. (2009). Mixed reality systems. International Journal of Online Engineering (iJOE), 5(S2).
  22. Muller, D., Chilliischi, A., & Langer, S. (2012). Integrating immersive 3D worlds and real lab equipment for teaching mechatronics. Paper presented at the Remote Engineering and Virtual Instrumentation (REV), 2012 9th International Conference on.Google Scholar
  23. Norman, D. A. (2010). Natural user interfaces are not natural. Interactions, 17(3), 6–10. Scholar
  24. Odeh, S., Abu Shanab, S., Anabtawi, M., & Hodrob, R. (2013). A remote engineering lab based on augmented reality for teaching electronics. International Journal of Online Engineering (iJOE), 9(S5), 61. Scholar
  25. Ohta, Y., & Tamura, H. (2014). Mixed reality: Merging real and virtual worlds: Springer Publishing Company, Incorporated.Google Scholar
  26. Olalde Azkorreta, K., & Olmedo Rodríguez, H. (2014). Augmented reality applications in the engineering environment. In P. Zaphiris & A. Ioannou (Eds.), Learning and collaboration technologies. Technology-rich environments for learning and collaboration: First international conference, LCT 2014, held as part of HCI international 2014, Heraklion, Crete, Greece, June 22–27, 2014, proceedings, part II (pp. 83–90). Cham: Springer International Publishing.Google Scholar
  27. Oshita, M., & Matsunaga, T. (2010). Automatic learning of gesture recognition model using SOM and SVM. In G. Bebis, R. Boyle, B. Parvin, D. Koracin, R. Chung, R. Hammoud, M. Hussain, T. Kar-Han, R. Crawfis, D. Thalmann, D. Kao, & L. Avila (Eds.), Advances in visual computing (Vol. 6453, pp. 751–759). Berlin: Springer.CrossRefGoogle Scholar
  28. Popa, D., Gui, V., & Otesteanu, M. (2015). Real-time finger tracking with improved performance in video sequences with motion blur. Paper presented at the 2015 38th International Conference on Telecommunications and Signal Processing (TSP).Google Scholar
  29. Quintas, M. R., Restivo, M. T., Rodrigues, J., & Ubaldo, P. (2013). Let’s use haptics! International Journal of Online Engineering.Google Scholar
  30. Reese, R., & Johnson, J. (2015). jMonkeyEngine 3.0 game development: A practical guide. Oakamoor: P8Tech.Google Scholar
  31. Restivo, M. T., & Cardoso, A. (2013). Exploring online experimentation. International Journal of Online Engineering, 9, 4–6.CrossRefGoogle Scholar
  32. Restivo, M. T., Rodrigues, J., & Chouzal, M. de Fàtima. (2014a, 3-6 December). Let’s work with AR in DC circuits. Paper presented at the 2014 International Conference on Interactive Collaborative Learning (ICL).Google Scholar
  33. Restivo, M. T., Chouzal, F., Rodrigues, J., Menezes, P., Patrão, B., & Lopes, J. B. (2014b). Augmented reality in electrical fundamentals. International Journal of Online Engineering, 10(6), 68–72.CrossRefGoogle Scholar
  34. Restivo, M. T., Chouzal, M. d. F., Rodrigues, J., Menezes, P., Patrão, B., & Lopes, J. B. (2014c). Augmented reality in electrical fundamentals. International Journal of Online Engineering (iJOE), 10(6), 68. Scholar
  35. Restivo, M. T., Quintas, M. R., Rodrigues, J., & Cardoso, A. (2015). Next-generation experimental lab #1. Paper presented at the 2015 3rd Experiment International Conference (’15).Google Scholar
  36. Smith, M., Maiti, A., Maxwell, A. D., & Kist, A. A. (2016). Augmented and mixed reality features and tools for remote laboratory experiments. International Journal of Online Engineering (iJOE), 12(07), 45. Scholar
  37. State, A., Livingston, M. A., Garrett, W. F., Hirota, G., Whitton, M. C., Pisano, E. D., & Fuchs, H. (1996). Technologies for augmented reality systems (Realizing ultrasound-guided needle biopsies). Paper presented at the 23rd annual conference on computer graphic and interactive techniques.Google Scholar
  38. Weichert, F., Bachmann, D., Rudak, B., & Fisseler, D. (2013). Analysis of the accuracy and robustness of the leap motion controller. Sensors, 13(5), 6380.CrossRefGoogle Scholar
  39. Wu, H.-K., Lee, S. W.-Y., Chang, H.-Y., & Liang, J.-C. (2013). Current status, opportunities and challenges of augmented reality in education. Computers & Education, 62, 41–49. Scholar
  40. Zhang, Z., Zhang, M., Chang, Y., Esche, S. K., & Chassapis, C. (2013). Integration of physical devices into game-based virtual reality. iJOE, 9(5), 25–38.Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Ananda Maiti
    • 1
  • Mark Smith
    • 1
  • Andrew D. Maxwell
    • 1
  • Alexander A. Kist
    • 1
    Email author
  1. 1.University of Southern QueenslandToowoombaAustralia

Personalised recommendations