Keywords

1 Introduction

The global economy is being rapidly transformed by sophisticated robots that enhance human dexterity, vision, speed, and strength. Meanwhile, advancements in automation technologies and Artificial Intelligence (AI) are changing how we view the future of work. These technologies are bringing disruptive changes to the employment market, requiring new skills and training in order to succeed in the future economy. Thus, preparing the workforce for an environment that is increasingly defined by these technologies is imperative.

Architecture, Engineering, and Construction (AEC) industries drive the design, construction, and operation of the built environment, affecting nearly every industry, trade, labor, and employment market across the global economy. However, despite their critical role in the job markets, the AEC industry is not ready to prepare their workforce for meeting the critical demands for a technically advanced workforce. Most buildings and infrastructure continue to be built on-site using centuries-old materials and techniques with labor construction productivity rates that are lower now than those reported over 50 years ago [11]. The training and apprenticeship model in the building trades has also remained largely restricted to on-the-job and on-site training with little emphasis on off-site and virtual training. Addressing the training needs of the AEC students and professionals is critical for ensuring the competitiveness of a large portion of the building industry workforce.

This paper outlines the process of developing an online robotics training system. The Robotics Academy, which is a game-based Virtual Reality (VR) learning platform, is designed to support the training needs of the AEC workforce. The Robotics Academy builds on research analyzing the AEC industry needs and leverages technological advances in adaptive, immersive, and data-driven systems to deliver a simulation platform for learning industrial robotics. Developing the platform draws on well-grounded learning theories to provide an innovative, effective, and personalized training environment.

The research and design of the Robotic Academy have been supported by two grants from the National Science Foundation (NSF). The initial grant supported the efforts of an interdisciplinary team of faculty for creating a roadmap for the Robotics Academy and developing a prototype VR learning environment, which was completed in 2020. The second grant, which supports the project until 2025, builds on the prior work to expand the Robotics Academy by further research and development of the prototype and integration of AI into the platform.

In the following sections of this paper, we will describe the theoretical foundation for developing the Robotics Academy and the implications of advancements in AI on creating virtual learning environments. We will then report on the completed research, describe various components and development activities, and conclude by stating the future research plans.

2 Learning Theories and Emerging Technologies

Emerging technologies have the potential to transform the way we learn. As a result, there has been a renewed interest in learning theories that inform the design of technology-mediated learning. As a framework for designing effective learning environments using advanced technologies, the constructivist theory has received considerable attention among many possible perspectives. The constructivist theory of learning was derived from the cognitive development research of psychologists Jean Piaget and Lev Vygotsky, who argued that individuals actively construct knowledge based on experience; therefore, knowledge cannot simply be handed down from one teacher to the next but rather must be constructed individually by each learner [12]. Learning Scientist David H. Jonassen states that constructivists are concerned with how learners construct knowledge, arguing that knowledge is a function of prior experiences, mental operations, and cultural constructs that help us to interpret events. He states that “constructivism views reality in the mind of the knower, that the knower constructs reality, or at least interprets it, based upon his or her appreciations” [19].

The constructivist approach to conceptualizing how we learn has several implications for our approach to developing technology-mediated environments. The idea that knowledge is a function of prior experience suggests that people develop knowledge and meaning from their experiences [1]. Therefore, the pedagogical approaches from a constructivist perspective focus on developing learner-centered, explorative, and task-oriented activities that engage the learner with interactive experiences [30]. Examples of this approach include Project-Based Learning, Game-based Learning, Situated Learning, Active Learning, and Experiential Learning. In these approaches, learners are considered the protagonists of the learning process. The learning environment is designed as a scaffolding composed of various scenarios geared to stimulate higher-level thinking skills, focusing on real-world problems [14].

2.1 Game-Based VR Learning

The constructivist theory emphasizes learner-centered experiences and activities. The literature on learning games indicates that the procedural and interactive nature of games provides an ideal mechanism for designing a constructivist curriculum based on exploration [3]. Additionally, immersive media, including VR, expands the capabilities of game engines [2], making game-based learning in VR a new research frontier. It has also been shown that immersive VR games can provide context and motivation for situated practice [6] and improve task performance [4, 25] through carefully designed game mechanics [9, 10].

Considering increasing evidence that games can motivate learners to persist in challenging tasks [16, 15, 27], engender high levels of cognitive, affective, and behavioral engagement [22], and destigmatize failure [23, 24], constructivist principles can be implemented more effectively. Specifically, virtual reality games can provide context for situated practice [25] through the careful use of game theory and the development of rigorous levels and missions [26, 27].

Design strategies for game-based VR learning include several attributes that support embodied and personalized learning. Immersive learning environments situate the learner in a sensory-rich environment and foster a sense of presence that can contextualize learning in various realistic settings, supporting situated cognition [7]. Games also facilitate simulated interactions as well as more complex and implicit cognitive engagement [29].

2.2 Adaptive Learning Systems

Constructivist learning theory also places emphasis on the individual's background and prior experiences, which prompts another set of reflections regarding the design of pedagogical approaches for technology-mediated learning [25]. According to Radianti et al. learners link new information to their prior knowledge, thereby rendering their mental representations subjective [24]. Due to this, learning environments are most effective when personalized to the individual learner's experience, aptitude, strength, and weakness [28].

Adaptive Learning Systems (ALS) are computerized learning systems that adjust learning content based on learners’ profiles, performance, and human factors [18]. Although ALS have been developed and tested in the last decades, the recent advances in AI and data-driven approaches render ALS more powerful. With the application of AI, learning content can be personalized based on an individual's prior experience, performance, and proclivities. AI-powered (ALS) are increasingly able to monitor learner performance and provide personalized instruction by adapting content, tasks, and feedback for individual learning and knowledge levels.

Additionally, advances in software and hardware which have led to state-of-the-art AR and VR Head-Mounted Displays (HMD), offer the possibility to collect and harness biometric data, such as learner movement, haptic interactions, eye-gaze, and heart rate variability. Access to this data enables researchers to analyze information correlated to attention, stress, and confidence levels for improving learning and training outcomes. The ability to collect and process this type of granular data from learners has significantly improved ALS.

3 Robotics Academy

Robotics Academy is an immersive online platform designed to address the needs of students and professionals in the AEC industries. Inspired by the recent technological achievements in self-adaptive, data-driven, and autonomous systems for virtual training, the platform integrates AI, VR, and big data analytics to create a personalized experience for industrial robotics training.

In developing the Robotics Academy, we applied three main strategies. The first strategy was to research the challenges of the industry with automation and gather insights on their training needs. This was achieved by interviewing the AEC industry representatives and holding a summit to reflect on the significant industry trends and issues. This step helped us to better understand their needs for creating the appropriate curriculum, content, and the most effective delivery modality. The findings from the interviews and summit are presented in Sect. 3.1, and the resulting curriculum is discussed in Sect. 3.2.

The second strategy was to build on the principles of the Constructivist theory by developing a game-based VR learning environment that supports active and engaged learning. To achieve this goal, we have developed the learning content as a series of task and driven activities. This approach supports active engagement and exploration, as learners can conduct various tasks safely and repeat them in various contexts until they learn. This first version of the learning environment and curriculum has been developed and tested in a prototype as described in Sect. 3.3.

The third strategy was personalizing the learning process, another core concept of Constructivist theory. To achieve this goal, we are developing an ALS, which utilizes Machine Learning (ML) and Natural Language Processing (NLP) to process learners’ data to build a comprehensive picture of the learner. The system learns from the activities and performance of the learner by dynamically analyzing learning preferences, skill levels, progress, biometric information, and interaction with the system. Finally, the ALS develops a recommended path through the learning content and provides feedback on the learners’ activities. The development process of ALS is described in Sect. 3.4.

3.1 Interviews and Industry Summit

We began the project by conducting a series of interviews with AEC employers and employees to help us understand the impact of robotics and the training needs of the industry. These included eighteen one-hour-long, semi-structured interviews with industrial roboticists, software developers, automation engineers, system integrators, AEC educators, and product design specialists. We also interviewed analogous employees from car manufacturing and product inspection companies. Our questions for employers focused on changes in their employing pattern, jobs shifted or created, new roles and skills, industry trends, workflows, and employee training protocols.

Major conclusions drawn from the employer interviews were: (1) robotic automation is rapidly moving to take over jobs in 3D areas (dirty, dull, and dangerous); (2) new applications for robotics automation, particularly industrial robotic arms, are rapidly increasing because of their versatility and efficiency; (3) demand for skilled workers, especially those who can use new technologies and equipment, is high, but they are in short supply globally; and (4) the adoption of automation and robotics in the building industry is creating new jobs, entirely new job classes, and opportunities for entrepreneurship.

Questions for employees focused on their educational background and experience, their on-the-job training, learning preferences, training challenges, and awareness of future technologies. Key conclusions include: (1) employees do not have access to adequate training materials and they mostly learn through trial and error; (2) the use of online sources and manufacturers’ manuals to run machinery, particularly robotic arms, is inefficient and time-consuming; (3) the lack of a standard or unified robotic design and manufacturing process is a major impediment to the learning process; therefore, the knowledge pool is scattered and not comprehensive; (4) employees need some level of in-person training and interaction with robots; (5) certificates and credentials leads to confidence, ability to showcase new skills, and job security.

These interviews confirmed some of our initial assumptions about the increasing rate of adoption of robotics in the industry as well as the scarcity of training materials. More importantly, it helped us develop the curriculum based on what was considered necessary for robotics training by the people actively engaged in its implementation and use.

In addition to interviews, we conducted a summit to gather representatives from the AEC fields, including the region’s leading AEC firms, two of the nation’s largest housing builders and developers, as well as four international experts and educators. The Summit highlighted several concerns: (1) the AEC industry faces a growing labor shortage; (2) the construction industry must dramatically improve its productivity and building performance to meet demand; (3) a transition to automated design, engineering, and construction will have severely disruptive impacts on the AEC sector with its massive potential for increased productivity leading to new jobs in robotic programming and factory-based prefabrication.

3.2 Curriculum

To design the training curriculum, we studied existing in-person and online robotics training curricula, including Kuka College, Universal Robots Academy, and ABB University. We also drew on our interview and summit findings. The resulting curriculum is composed of six modules, each containing several activities. The introductory modules focus on the fundamentals of industrial robotics, including robotic anatomy, safety, movement, and calibration. More advanced modules focus on programming, end-effector design, and integration. Each module has brief instructions and a series of tasks delivered as games with a scoring system. The modules are designed as independent and hyper-granular concepts to allow quick reorganization of the concepts by the ALS. This will allow quick reshuffling of the content based on the learner’s performance.

3.3 Prototype Development

The design and deployment of the game-based VR prototype began with training students with KUKA (KR10 and KR30) and Universal Robots (UR10). These robotic arms were used to establish and test learner interactions for our game (Figs. 1, 2 and 3). The VR prototype was developed in the Unity game engine for the HTC Vive Eye head-mounted display (HMD). The steps for creating the prototype included:

  • Defining learning objectives: This involved determining the knowledge and skills that learners should acquire through the VR learning experience, and the types of challenges and interactions that would help achieve these objectives.

  • Understanding curriculum content for VR adaptation: We observed a user group engaging with the curriculum in person to learn about the process and evaluate best methods to adapt the curriculum for VR.

  • Developing a Game Design Document (GDD): This entailed creating a guide describing the premise, gameplay, art direction, engineering, sound, and music.

  • Converting GDD to AGILE software development: This allowed us to develop strategies for team management, enabling more efficient delivery and greater responsiveness to change.

  • Designing and developing the main VR features: We included interactions related to vision, including eye-gaze tracking, as well as sounds and touch, to understand how UI and UX would be uniquely developed.

Fig. 1
figure 1

Left: Scene from the VR learning prototype demonstrating lesson selection process. Right: Simulation in the Pick and Place lesson

Full size image
Fig. 2
figure 2

Scenes from the VR learning prototype. Top: This lesson challenges the students to match the position and orientation of a “ghosted” robotic arm using both Axis-specific and World-specific motion systems. Bottom: This lesson focuses on various movements

Full size image

The integration of the resulting work from these steps led to the completion of our first prototype. The next version of the prototype, which is currently underway, includes programming the game mechanics, building a procedurally generated robotic training space, improving biometric data collection method from the VR HMD, and implementation of the Adaptive Learning System.

Fig. 3
figure 3

Scenes from the VR learning prototype demonstrating the learner dashboard including biometric analysis and further progress information

Full size image

3.4 Personalizing the Learning Environment

One aspect of the Constructivist Learning theory is the central role of the learner’s prior knowledge in constructing new knowledge. ALS has the potential to personalize the learning experience by responding dynamically to the individual learner’s strengths, weaknesses, and proclivities. Thus, personalization of the learning content based on the learner's previous experience is a critical element in developing personalized learning environments.

To develop the capacity for personalizing the learning content for the individual learner, we needed to understand the individual learner in a holistic way. This required collection of several data sets to closely monitor learner progress which were: (1) Learner Profile Data including information about age, gender, ethnicity, preferred language, academic background, and skill sets in robotics, and (2) Biometric data including eye gaze and haptic interaction, (3) and Performance Data, including game interaction (task completion time, number of attempts and errors, and test and quiz scores.

Although numerous approaches for developing ALS have been described in the literature, in general they are composed of three interdependent models governed by an Adaptive Engine. These include a Learner Model, a Domain Model, and an Instructional Model [21]. Building on this established research, our project’s ALS system models include:

  • Learner Model: The Learner Model records learners’ personal characteristics and reflects learners’ individual differences, supporting the decision-making process for the ALS [3]. Currently, the analysis and integration of this data is underway by our computer science team. Using ML and NLP, they are working towards developing a comprehensive picture of the learner. The Learner Model will incorporate collected demographic and prior experience information for establishing the learner’s characteristics.

  • Domain Model: This model consists of the curriculum, learning objectives, and learning content. We have built the learning content based on our interview findings to respond to AEC industry needs. The Robotics Academy curriculum has several modules on safety, robotic anatomy, kinematics, movement programming, end-of-arm tooling, and robotic simulation. This model leverages the predefined rules and functions provided in the Instructional Model to intelligently and dynamically select the most suitable learning content and tasks, ensuring a personalized learning experience that takes into account the learner's unique profile, progress, and preferences.

  • Instructional Model: This model refers to the algorithm that guides instruction based on both the Domain and Learner Models. It serves as the basis for making decisions regarding learning content delivery, including what, when, and how adaptation should occur [5]. Our Instructional Model uses rubrics for learning assessment based on periodic quizzes and tests, measurements of task completion times and error detection, as well as estimated level of learner focus, and stress based on eye gaze tracking data.

  • Adaptive Engine: Using ML, the Adaptive Engine will correlate data from these three models to trace real-time learner interactions with the VR-Game, continuously match task performance with the Instructional Model rubrics to measure learning outcomes. Meanwhile it will guide teaching strategies by selecting learning content from the Domain Model and testing it against learner performance data. Finally, it will provide targeted feedback including relevant explanations, hints, examples, demonstrations, and additional or alternative tasks for the individual learner.

The addition of ALS to the VR learning system promises to expand the potential of the Robotics Academy as a learning tool for industrial robotics training (Fig. 4).

Fig. 4
figure 4

Robotics Academy’s Adaptive Learning Systems for Personalizing Learning in VR

Full size image

4 Conclusion and Future Plans

Initial testing of the Robotics Academy VR learning game prototype suggests that it is at least as effective as an in-person training course for teaching the fundamentals of working with industrial robots [23]. Furthermore, analysis of testing data shows that the platform can produce comparable, quantifiable learning outcomes to traditional settings and has the potential to serve as an effective replacement for traditional robotics training.

Further testing data analysis shows that learners who were allowed to practice specific tasks in the immersive environment could transfer VR-acquired skills to operate a physical robotics arm [23]. This reinforces our hypothesis that learning in simulated environments can support skill development for real-world applications in robotics. Additionally, the application of devised ALS promises to significantly expand the potential of the project as a learning tool. The two NSF grants have provided the opportunity to conduct foundational research, develop a plan for the Robotics Academy and create a VR learning game for training. In the following months, we plan to expand the capabilities of our prototype through the entire development and implementation of the ALS.

Since ALS requires large-scale data to function well, expanding usage and testing are critical for the success and scalability of this project. Therefore, we plan to test the Robotics Academy with 100 students, which will provide a sufficient dataset for iterative improvement and model training. The development of this learning tool will provide valuable insights into the learning process for researchers and developers studying immersive learning environments. The project will shed new light on integrating ML and NLP to develop effective ALS. Our goal is to increase opportunities for automating education and training in the AEC industries.