1 Introduction and Motivation

In recent years, Artificial Intelligence (AI) and Robotics have become major topics with a huge impact not only on our everyday life but also on the working environment and the society itself [47]. Nowadays, almost everybody uses or is affected by AI and Robotics in daily life. Having knowledge on the principles and concepts of these technologies empowers people to take well-informed decisions and to deal with issues like deep fakes, fake news or micro targeting. Most areas of life and occupational fields are already related to AI and Robotics. Since this trend is set to continue in the near future, there is a strong demand for well-trained people in these areas. Hence, sound knowledge on the principles and concepts of AI and Robotics are key skills for the 21st century and essential for future careers in science and engineering [66]. It is important to teach fundamental AI and Robotics concepts already prior to, or outside of university in order to (a) address the serious lack of qualified staff in technical professions many countries are already confronted with, and (b) to foster a solid understanding of sociopolitical, economical and technical aspects of AI and Robotics.

The speed with which AI and Robotics are impacting our everyday life as well as our working world poses a tremendous challenge for our society and educational systems. Familiarizing young people with these topics, fostering AI/Robotics literacy and therefore allowing them a social and economic participation are crucial and challenging aspects for educational systems all over the world. However, fundamental AI/Robotics concepts and techniques have traditionally been taught almost exclusively at the university level. Hardly any systematic, well-grounded approaches exist that focus on teaching principles of intelligent systems at K-12 level, addressing both teachers and school students. Moreover, these topics are almost completely absent from most school curricula.

In order to prepare and motivate young people to engage in AI/Robotics related careers and also to provide the benefits of AI/Robotics education for all, first, the teachers and educators have to get an adequate training. These teachers then act as multipliers, directly addressing and educating their students [47].

In order to meet these challenges, the European Driving License for Robots and Intelligent Systems (EDLRIS) was developed [39, 41, 49]Footnote 1. This novel educational project aimed at the development and implementation of a professional, standardized, and internationally accepted system for training and certifying teachers and educators (in this context referred to as trainers) and school students, apprentices and young people (referred to as trainees, learners respectively) in AI and Robotics. It comprises distinct training modules for beginners and advanced learners, following a train-the-trainer, competency-based, blended-learning approach. A certification system proves trainers’ and trainees’ skills acquired during the training. All modules have been implemented and evaluated between the second quarter of 2019 and the first quarter of 2020, mainly in Austria and Hungary. Further implementations are rolled-out continuously. The entire approach is based on a number of prior projects which focused on teaching fundamental concepts of AI and Robotics to teachers and young people of different age groups, ranging from kindergarten kids to high school and undergraduate students [15, 25, 40, 42, 44]. Educational concepts to teach basic Robotics and AI principles to young school students and teachers were developed and tested in [25, 40, 44], whereas concepts to teach more advanced Robotics and AI principles to older school students were developed and evaluated in [15, 42].

The remainder of this paper is structured as follows: Sect. 2 discusses related literature, Sect. 3 presents the applied methodology and project approach, Sect. 4 deals with the implementations and evaluations and Sect. 5 summarizes the contribution of this article.

2 Related Literature

The EDLRIS project was inspired by the well-established European / International Computer Driving License (ECDL/ICDL), a program for training people and certifying their skills in common computer applications (e.g. text processing, spreadsheet) [19]. A train-the-trainer and certification concept was applied by the Roberta Initiative [13], but it specifically focused on raising young women’s interest in technology. Using robots as educational tool (also referred to as educational robotics) and introducing selected robotics topics mostly at a beginner level has become a widespread approach in recent years [4, 8, 16, 56].

In contrast, teaching a broad spectrum of fundamental AI and Robotics concepts still happens, for the most part, exclusively at graduate or undergraduate university level [7, 9, 43, 60, 64, 65, 74]. A number of authors describe frameworks and games applied to teach university students fundamental AI techniques (e.g. using a Pacman framework [20, 46]). In addition, certain teacher education trainings at university level also focus on imparting basics of AI to teacher training students [23]. At university level several advanced courses and curricula on intelligent systems are offered at technical universities [67]. The topics covered correspond to the findings of Wollowski et al. [72] with knowledge representation and reasoning at the heart of the curricula [44].

Several projects at K-12 level deal with teaching only selected topics like neural networks, history of AI, Turing test or chat bots [30, 35]. Other projects focus on specific tools to demonstrate certain AI/Robotics aspects [18, 28, 37, 73]. Various approaches teach selected computer science concepts and programming paradigms, but not specifically focusing on AI [45, 59].

However, in recent years several initiatives and projects stressing the importance of a systematic AI education at K-12 level have emerged [2, 47, 66, 71]. In this context, also education organizations and AI experts as well as governments started to develop and deploy AI curricula and programs for a K-12 audience [27, 31, 58]. In addition, recent workshops and symposia also addressed the topic of teaching AI with special focus on a pre-university audience (AI in K12 symposium [1]; EAAI symposium [24]; EduAI workshop [26]; AIED4K12 workshop [3]).

Compared to the discussed approaches, the project presented in this article follows a novel and unique concept: A broad spectrum of fundamental AI and Robotics topics are taught to both teachers (acting as multipliers) as well as to school students and young people on a beginner and an advanced level. The EDLRIS project does not focus solely on online courses but combines face-to-face and online units, applying a blended-learning approach. Based on a combination of well-grounded learning theories and tools as well as methods with a strong focus on hands-on activities, technically and didactically sound curricula and teaching materials were developed. Complex topics, usually taught at university level (e.g. symbolic AI using declarative programming, kinematics and path planning in Robotics, etc.), were prepared for teachers and school students at K-12 level (high school), yet still focusing on providing students and teachers with a deep understanding and fostering a scientific problem solving approach (e.g. analyze, describe, model and solve problems). Additionally, a standardized certification system proves the acquired skills of teachers and students after completing the training. This unified approach enables people to take informed decisions about AI and Robotics (e.g. knowledge of the technical concepts of a smart assistant is crucial for the decision whether to use this technology or not) and allows young people a social and economic participation.

3 Methodology

3.1 Project Approach

Keeping these challenges in mind, the main goals of EDLRIS could be summarized as follows:

  • training teachers, mentors, pedagogues and educators (= trainers) as well as secondary school students, apprentices and young people (= trainees) in basic and advanced topics and principles of AI and Robotics

  • providing training curricula and adequate, well-grounded technological and didactical teaching material

  • conducting face-to-face and online training courses for trainers and trainees (= training system)

  • testing and certifying AI/Robotics skills of trainers and trainees acquired during the training (= certification system)

  • ensuring that these competencies and skills are also recognized by educational institutions and industry

  • ensuring sustainability as well as a high acceptance and an international deployment of the training and certification system

In general, the project comprises four stages [41]:

Stage 1—foundation: The EDLRIS project consortium is composed of two technical universities in Austria (AT) and Hungary (HU) which have a strong research background in AI and Robotics ensuring sound technological preparation, one university of teacher education (AT) ensuring the sound didactical preparation as well as two computer societies (AT, HU) with long-term experience in conducting computer certifications across Europe. As a first step, the educational needs of society and economy as well as the expected Robotics and AI skills of graduates were assessed by conducting a survey among various stakeholders from educational institutions and industry. The survey was done as a series of semi-structured qualitative interviews which were audio-recorded and subsequently transcribed [44, 50].

In order to ensure sustainability and a broad acceptance of the training and certification system, an advisory board comprising representatives from educational institutions, public organizations and industry (project and human resource managers, representatives from the ministry of education, the chamber of commerce and private/public institutions) was established. The advisory board was involved from project start to the completion of the project (annual meetings and workshops, continuous integration and feedback, final stakeholder workshop).

Stage 2—system development: Based on workshops with members of the advisory board, the survey results (stage 1) and aligned with Austrian and Hungarian educational curricula the main learning objectives of the training curricula were defined. According to these national curricula for computer science, school students should be able to describe, design, present and implement algorithms and be able to explain the basic principles of automata, algorithms, data structures and data modeling. School students should learn about basic concepts associated with automated development systems and basic concepts in robot control. In addition, the curricula for the elective subject of computer science dedicate parts to the discussion of intelligent systems (characteristics, areas, use-cases, opportunities, threats) [14, 44].

The advisory board together with experts in AI, Robotics and education collaborated on the development of the curricula and learning objectives, following well-established procedures [44, 53]. Based on these learning objectives, the core competencies and main topics for four training modules were derived: AI basic and advanced (for trainers and trainees) as well as Robotics basic and advanced (for trainers and trainees). Subsequently, the training system comprising curricula and course materials, addressing both beginners and advanced trainers and trainees, was developed. The development of the certification system was accomplished simultaneously. The certification system consists of tested certification questions (multiple-choice questions (MCQ) and interactive tasks) for each of the four modules to prove trainers’ and trainees’ understanding of AI and Robotics topics (also see Sect. 3.3; Fig. 7 shows an example of an interactive certification task). Training and certification systems are integrated into a unified online learning and course management system [17]Footnote 2.

Stage 3—train the trainers: The initial trainings for trainers (teachers, mentors, educators) are conducted by researchers of the project consortium following a train-the-trainer, blended learning approach [7, 11]. The four modules can be completed separately and comprise several face-to-face (class attendance) and online teaching units. After successfully completing the training and the certification, trainers obtain a widely recognized EDLRIS trainer certificate.

Stage 4—train the trainees: Certified trainers are then entitled to conduct trainee courses. They therefore act as multipliers by training trainees (school students, apprentices, young people) in accordance with the training curricula and course materials developed at stage 2. The trainee trainings for each of the four modules also combine face-to-face and online teaching units following a blended learning, student centered approach. After successfully completing the training and the certification, trainees receive an EDLRIS trainee certificate. It is important to mention that only people who successfully completed a trainer course are entitled to teach trainees. While the content-related parts of trainer and trainee courses are identical, trainer courses additionally include didactical aspects (teaching methods, tools, learning theory).

The described approach is iterative: While continuously conducting trainer and trainee courses and certifications, feedback and evaluation results are used to improve, debug and extend the training and certification system (see Fig. 1).

Fig. 1
figure 1

Overview of the EDLRIS project approach

Full size image

3.1.1 Core Competences

This paragraph provides an overview of the core competences of each moduleFootnote 3.

AI basic:

  1. 1.

    I am able to describe AI, to recognize AI systems and to distinguish AI systems from other concepts and systems.

  2. 2.

    I know the areas of application of AI and their use cases and I am aware of the technical, social, ethical and legal implications.

  3. 3.

    I am able to formalize a problem and to apply algorithms and data structures to solve this problem.

  4. 4.

    I am able to design and practically implement a very simple AI system for a given application.

AI advanced:

  1. 1.

    I am familiar with different AI areas and frameworks and I am aware of ethical, social and legal implications of AI systems.

  2. 2.

    I can master the required mathematical basics and I am able to understand and describe basic AI concepts.

  3. 3.

    I am able to describe problems, which require an AI-related solution, in a formal way, and furthermore, I am able to solve these problems by applying adequate algorithms.

  4. 4.

    I know the fundamental properties of problems, representations and algorithms.

  5. 5.

    I am able to analyze, configure, maintain and integrate an existing AI tool and I am able to systematically design and practically implement a simple AI system for a given application.

Robotics basic:

  1. 1.

    I am familiar with the history, the background, the terminology and the fields of application of Robotics and its use-cases and I know about the social, ethical and legal implications.

  2. 2.

    I understand the big picture of robot system i.e. the context and the ecosystem where the robot is integrated.

  3. 3.

    I understand the potential of Robotics and I am creative in imagining new scenarios and use-cases.

  4. 4.

    I know and understand the fundamental Robotics concepts.

  5. 5.

    I know the components required to implement the fundamental Robotics concepts.

  6. 6.

    I am able to integrate components in a Robotics system for a simple task.

Robotics advanced:

  1. 1.

    I can master the required basics of mathematics, physics and computer science.

  2. 2.

    I know about and I am able to apply a fundamental systematic engineering approach.

  3. 3.

    I have knowledge of the fundamental mechanical, electronic, algorithmic and computer science concepts and I am able to apply appropriate tools and methods required to configure and to implement a Robotics system.

  4. 4.

    I am able to works with real life equipment (robots).

  5. 5.

    I am aware of ethical and social aspects in the context of Robotics.

3.2 Training Modules

Respecting established rules of curriculum design, the main learning objectives and competences of each training module were defined and core topics derived (stage 2) [53]. Subsequently, detailed contents and schedules as well as course and teaching materials for the training modules were developed. Since research in education has proven the positive learning effects of hands-on experiences, this development was based on well-grounded concepts and principles of constructionism [33, 54, 55]. Constructionism emphasises the active part of the learners in the learning process. Therefore, each module comprises hands-on activities and practical exercises (e.g. constructing a simple mobile robot, working together in small groups to implement collaborative micro-projects like an AI-based pac-man game or an indoor-navigation robot).

In general, the entire project follows a blended learning, competency-based, student centered approach [32]. The teaching materials comprise ready-to-use tasks, scripts, tutorials, slides, tools (software and hardware) as well as examples and hands-on exercises, including sample solutions for trainers. Each module applies a broad range of various learning methods and teaching concepts, among others, problem-based learning, project-based learning, discovery learning, inquiry learning, collaborative learning, teamwork, flipped classroom, active plenum, attentive micro lecture as well as whole brain teaching [5, 9,10,11, 34, 52].

In addition, a variety of learning tools are used in each module, such as, computer science unplugged, educational robotics, programming exercises, simulators as well as puzzles (e.g. ABC-graffiti method) and educational games [4, 8, 29, 61]. The teaching and course materials for each module as well as the certification system are integrated into Moodle, a widely used online learning management system [17]. By providing online teaching material and offering the possibility to complete the certification online, access to the training and certification system is ensured even in remote areas.

The basic AI and Robotics modules specifically focus on teachers and high-school (secondary school) students without any prior knowledge in these areas. The goal is to build awareness, to motivate and to introduce principles of AI and Robotics in an easily accessible manner in order to be able to live and work with the technology and to allow an educated discussion. A basic module comprises 24-32 hours (3-4 days) of face-to-face and approximately 30 hours of online units (depending on the individual learning pace).

The advanced AI and Robotics modules are primarily aimed at teachers and high-school (secondary school) students who already have prior knowledge in computer science, mathematics, and physics. The goal is to enable a deeper understanding, to understand fundamental ideas and to foster people’s ability to implement simple AI and Robotics applications. An advanced module comprises 32-40 hours (4-5 days) of face-to-face and approximately 50 hours of online units (depending on the individual learning pace).

The applied blended learning approach is a combination of traditional face-to-face and online units [11]. As recommended in literature, blocks of face-to-face units are followed by online units [70]. The units can also be arranged in a modular way, ensuring a flexible course structure (see Fig. 2).

Fig. 2
figure 2

Three possible ways of how to arrange an EDLRIS course, combining face-to-face (F2F) and online units

Full size image

Next to technical aspects, each module addresses the issue of ethical and social aspects of AI and Robotics and encourages people to think about and discuss implications of these technologies. In this context, discussions and interactive learning techniques like the World Cafe method [6] are applied. The modules comprise a broad range of practical activities and hands-on exercises to be solved in smaller groups. EDLRIS therefore fosters collaboration, communication, critical thinking and problem solving abilities, which are part of 21\(^{st}\) century skills [69].

Additionally, the trainings for trainers (stage 3) comprise discussions of applied teaching concepts and learning tools. Finally, each module concludes with the certification of trainers/trainees. The certification consists of an online, open-book exam with MCQ and interactive questions (also see Fig. 7).

The topics of each module were derived from the initially specified learning objectives and core competencesFootnote 4 and were aligned with national educational curricula (as described in Sect. 3.1) and with general AI and Robotics topics (as depicted in reference textbooks [62, 63]). The contents were then adapted and prepared with respect to the age- and target-group (K-12 students, teachers). The following sections provide an overview of the structure and the main topics/contents of each module.

3.2.1 Artificial Intelligence Basic

The module is structured around major topics and principles of AI as described in the reference textbook by Russell and Norvig [62]) and focuses on a playful teaching approach as proposed by Zyda and Koenig [74]. Preparatory online units introduce participants to Python programming and basic data structures. Further online units include exercises to repeat and consolidate already learnt knowledge. During the face-to-face units, participants practically implement a simple AI system for a given application (implementing a simple maze solver as described by DeNero and Klein [20] or a simple chat-bot).

In this context, participants learn about (mostly in face-to-face units): AI definitions and applications of AI in daily life; identifying and recognizing AI systems; fundamental data structures in AI (graph, stack, queue); basics of propositional logic; principles and applications of natural language processing, computer vision and machine learning; solving problems by search (depth-first search (DFS), breath-first search (BFS), greedy search); formalizing a problem and applying algorithms and data structures to solve this problem. Main teaching tools are unplugged exercises as well as Python tools, libraries and frameworks. Figure 3 illustrates an activity of this module.

Fig. 3
figure 3

AI basic: Playful introduction of problem solving by search (DFS, BFS, greedy) with an unplugged card game “find the Mona Lisa”. The graph on the right represents the floor plan of the Louvre (nodes represent rooms). The goal is to find the room where the famous painting is located by first playing the card game, and then to traverse the graph (a detailed description of this activity can be found in [44], p.73)

Full size image

3.2.2 Artificial Intelligence Advanced

The module is also structured around major AI topics and principles as described by Russell and Norvig [62] but having a closer connection to current AI research and covering topics in greater detail and complexity (as opposed to the AI basic module). Preparatory online units provide the necessary basics in calculus, linear algebra, probability theory, logic and Python programming. The face-to-face units focus on the three fundamental AI concepts: problem solving by search, declarative and data-driven AI concepts. In this context, participants implement different smaller AI projects and work with common AI frameworks and tools (Prolog, PDDL Editor, MiniZinc, TensorFlow).

By working on these tasks, participants learn about AI definitions, applications and history of AI, fundamental data structures and mathematical concepts of AI, properties of problem representations and algorithms, modeling and analyzing problems as well as principles of knowledge representation and reasoning, knowledge based systems, common-sense knowledge, classical planning, natural language processing, constraint satisfaction problems and machine learning. Further online units include exercises to repeat and consolidate already learnt knowledge. Main teaching tools are unplugged exercises (e.g. paper and pencil tasks and calculations, white board and flip chart activities, card games), simulators and the previously mentioned AI frameworks and tools. Figure 4 illustrates an activity of this module.

Fig. 4
figure 4

AI advanced: Introduction to classical planning using unplugged activities and PDDL (planning domain definition language)

Full size image

3.2.3 Robotics Basic

The module focuses on introducing the basic concepts of robotics using a constructivist educational robotics approach as proposed by Barker [8] and Alimisis [4]. The main goal of this module is to familiarize participants with the fundamental concepts of robotics and intelligent systems (sense-plan-act cycle) as well the components such as motors, sensors and controllers required to implement these fundamental Robotics concepts. In this context, participants work (mostly in face-to-face units) on three practical mini-projects with increasing complexity (domain: robotics in the context of autonomous driving).

By working on these task, participants learn about history, terminology, applications and use cases of robotics and robot systems in our everyday life. Furthermore, they are familiarized with the modeling of a robotics system (using flowchart modeling) and learn how to build and program mobile robots. Next to unplugged exercises, this module uses LEGO Mindstorms EV3 robots with the graphical programming language Microsoft MakeCode as the main educational robotics learning tool [21, 68]. Figure 5 illustrates an activity of this module.

Fig. 5
figure 5

Robotics basic: Fundamental robotics concepts and components (theory + practice with Mindstorms robots and graphical programming language)

Full size image

3.2.4 Robotics Advanced

The module has a close research connection and is structured around the main topics of robot manipulators (robotic arms) and mobile robots as described in the reference textbook by Siciliano and Khatib [63]. Preparatory online units provide the necessary basics in calculus, linear algebra and Python programming. During the subsequent face-to-face units, participants are given two concrete problems: 1) mathematical description/modeling of a certain robotic arm and its trajectory; 2) indoor localization and navigation of a mobile robot.

By working on these tasks, participants apply a systematic engineering approach and learn about the kinematical model of manipulators and mobile robots (direct/inverse kinematics, homogeneous transformation, DOF/DOM, Jacobian) as well as sensor fusion and state estimation (probabilistic model, Bayesian and Kalman filter). Teaching tools are unplugged exercises (e.g. paper and pencil tasks and calculations, white board and flip chart activities), simulators (Python) and the TurtleBot 2 robotics platform, a research robot commonly used in robotics education [12]. Figure 6 illustrates an activity of this module [49].

Fig. 6
figure 6

Robotics advanced (schematic diagrams): modeling of a robotic arm as well as indoor localization with TurtleBot [49]

Full size image

3.3 Evaluation Methodology

In order to evaluate the entire program, well-grounded quantitative and qualitative research methods have been applied [22, 36, 38, 48, 51]. A survey among trainers was administered prior and after each course (Likert scale and open-ended questions) in order to gather general feedback on the training system and content as well as specific feedback on participants’ assessment of the blended learning approach [57]. In addition to this survey, quantitative pre- and post-tests (conducted before and after training completion) were scheduled using a questionnaire with 10 multiple-choice knowledge questions in order to investigate learning effects. The learning gain was then calculated using inferential statistical procedures (paired sample t-test). Participant observations during the courses as well as qualitative interviews with a representative sample of participants provided additional information on the quality and impact of the training.

As described in Sect. 3.1, for each of the four AI/Robotics basic and advanced training modules a certification exam was developed. The certification exam comprises 18-20 multiple-choice questions and interactive tasks which cover the main topics of each module (Fig. 7 illustrates a certification task). General conclusions regarding the success of the training can be drawn by using descriptive statistics of training completion and certification.

The subsequent sections present and discuss the evaluation results.

Fig. 7
figure 7

Example of an AI advanced certification task (topic: problem solving by search using A* algorithm; the task is to assign the correct costs (f) for the corresponding cities)

Full size image

4 Implementation and Evaluation Results

At the time of writing (March 2020), the project successfully completed stage 1 (pre-survey, preparation) and stage 2 (learning objective definition, curriculum, teaching material and certification system development) as well as the initial implementations of stage 3 (train the trainers) and stage 4 (train the trainees). In general, the evaluation was carried out in two steps: A pilot implementation and evaluation of stage 2 as well as the first regular implementations and evaluations of stages 3-4. Respecting legal and ethical requirements all collected information was treated confidentially and personal data was anonymized.

4.1 Pilot Implementation

In order to test and evaluate the teaching approach and methodology in general, as well as the developed course materials and applied teaching methods/tools in particular, a first pilot class of the AI basic module was implemented in April 2018. The evaluation methods applied were participant observations, semi-structured interviews as well as a 3-part participant questionnaire (background information, Likert scale feedback on methods/tools/topics, self-evaluation of learning gain). All in all, 16 teachers (4 female, 12 male) participated in this first pilot test. After analyzing all gathered data the results showed that the competency-based, student centered approach is, in general, an appropriate method to achieve the educational goals of this project. It turned out that certain teaching methods, and teaching materials/tools had to be adapted, and the complexity and extend of certain contents had to be reduced in order to impart AI/Robotics topics in a target group specific manner. Based on the insights of this pilot implementation, the further AI and Robotics modules were developed and adapted accordingly [41]. The thesis of Lassnig [44] describes in detail the development, realization, and evaluation as well as results and conclusions of this first pilot implementation.

4.2 Regular Implementations

The first regular trainer and trainees courses have been implemented between the second quarter of 2019 and the first quarter of 2020. In sum, 450 people (trainers + trainees) participated. Out of the total number of participants, 57% were trainers (mainly science teachers from different secondary schools and educators from educational centres) and 43% trainees (mainly secondary school students from technical and non-technical high schools). 58% of the trainers successfully completed the certification, subsequently acting as multipliers by teaching trainees (i.e. in schools or science clubs)Footnote 5. Also 58% of the trainees successfully completed the certification. This is mainly due to the fact that many trainees completed the training but decided not take the final certification exam. Another reason for not passing was - according to teachers’ comments - that students did not prepare well for the exam. Overall, it is a satisfiable rate since we expected that half of the participants would successfully complete a certification (see Fig. 8). The completion ratio was determined as follows: Trainers had to reach at least 75%, trainees at least 50% during the final certification exam for a successful completion. In total, 32 courses were held, whereas each of the trainer and trainees module for AI and Robotics has been implemented at least once. Figure 9 provides an overview of module implementations as well as the participant and certification distribution along the modules.

Fig. 8
figure 8

Participants and certification rates

Full size image
Fig. 9
figure 9

Overview of implementations for each module (left); participant and certification distribution along the modules (right; n=445)

Full size image

The first train-the-trainer courses and certifications started in April 2019 and were finished by May 2019. Stage 4 then started in July 2019 with the first pilot implementations and evaluations of trainings and certifications for trainees (in the way of summer camps for high school students). After analyzing the evaluation data and adapting the training material, the trainee courses continued throughout the second half of 2019, the first quarter of 2020 respectively. These courses were implemented in schools by previously trained and certified trainers. In parallel, further trainer courses and certifications were conducted from September 2019 to March 2020. These highly qualified trainers will then again conduct trainings for trainees (students) in schools. In order to address the high demand as well as to ensure sustainability and a broad distribution, further trainer trainings will be conducted on an annual basis and certifications are offered throughout the year.

As described in Sect. 3.3, a combination of quantitative and qualitative research methods were applied in order to evaluate the program. Summing up the results of the survey among participants of the first train-the-trainer courses (n=52), 92% stated that their expectations towards the training were met and over 90% stated that the face-to-face units were essential for their learning success. On the contrary, only around 80% agreed that the online units were sufficiently aligned with the face-to-face unitsFootnote 6. Participants also mentioned that certain courses had too little time between face-to-face units (to foster learnt knowledge and to prepare for the subsequent unit).

The overall feedback regarding course structure, content preparation and applied teaching methods was positive. Nevertheless, participants mentioned that the gaps (in terms of complexity) between the basic and the advanced modules were too large. This would make it hard for trainees to fully understand the complex topics, and furthermore, to implement these advanced topics at school. This was a frequent statement regarding the more complex mathematical calculations in the Robotics advanced module. Another concern raised by the participants during the interviews was the cost factor: Many schools could not afford buying expensive robots for the Robotics basic and advanced modules. Further results of the survey, as well as the results of interviews and discussions with trainers during the courses revealed additional issues. Certain online units (in particular AI and Python) showed some content-related errors and were not intuitive enough for self-guided online learning sessions. Overall, the basic contents were rated as appropriate for trainees, but certain topics would still be too complex for students without any prior knowledge (e.g. PID controllers in Robotics basic or greedy search in AI basic). During the interviews participants also mentioned that the AI modules explain technical concepts using unplugged activities very well, but additional AI tools and programming tasks would be necessary. Furthermore, the curricula and teaching material developed would still need some additional preparation and improvement in order to be used in a more flexible way, taking into account the different requirements of different school types and settings [49, 57].

In addition to the survey, quantitative pre- and post-tests during the first two regular AI basic trainings for trainers and the first regular AI basic training for trainees were conducted using a questionnaire with 10 multiple-choice knowledge questions. These questions mainly dealt with technical concepts of AI (as described in Sect. 3.2.1) as well as one question regarding ethical aspects of AI. Data analysis (paired t-test) showed a statistically significant improvement between pre- and post-test (n=22; t(21)=18.086, p<.001; SD pre 5.59, SD post 4.28; see Fig. 10).

A certification exam (being another assessment and evaluation instrument) covered the main topics of each module. From analyzing the results of the exams, it could be seen that people understood the fundamental concepts and topics of AI and Robotics (as discussed in Sect. 3.2) quite well, but had problems with completing some of the more general Python programming tasks. This might be due to the fact that the main focus of the courses was on teaching the fundamental AI/Robotics concepts rather than teaching programming.

It has to be mentioned, that these are preliminary results only and that the sample sizes of the trainer survey and the pre-/post-tests are small. Further data analysis and additional quantitative evaluations of implementations of all four AI and Robotics modules - using a refined and more extensive questionnaire - are necessary to get a more conclusive picture and more significant results.

Nonetheless, the evaluation revealed the need for improvement regarding certain aspects of the program (i.e. partly too advanced and complex topics for the target group; textual errors of certain online units, cumbersome handling/user interface respectively; more flexible course structure/tools/teaching materials for school implementations would be necessary). It also includes - among other things - revising AI and Python online teaching units, closing the gap between basic and advanced modules (i.e. simplifying mathematical calculations and complex Robotics topics like inverse kinematics and Kalman filter), adding additional hands-on programming tasks to the AI modules and revising the teaching concept in order to use affordable Robotics platforms (addressing the cost issues raised by teachers).

Fig. 10
figure 10

Results of pre- and post-tests of AI basic participants (n=22)

Full size image

5 Summary and Conclusion

The main contribution of this paper is the presentation of the European Driving License for Robots and Intelligent Systems (EDLRIS). This innovative educational project focused on the development, implementation and evaluation of a standardized, professional, and internationally accepted and distributed training and certification system for trainers (e.g. teachers, educators, mentors) and trainees (e.g. secondary school students and young people) in the area of Artificial Intelligence (AI) and Robotics. Since these topics are influencing and will continue to change our society and shape our future, people need profound knowledge and skills in AI and Robotics. As part of this novel educational project, teaching curricula and training modules for AI and Robotics have been developed following a competency-based, student centered approach based on the principles of constructionism. A blended learning, train-the-trainer procedure ensures a broad distribution of the training system as these trainers act as multipliers by teaching trainees in accordance with the developed training curricula and course materials (e.g. integrating in regular school education). Additionally, an advisory board, comprising representatives of industry, public bodies and educational institutions was established and ensures a broad, international acceptance of the entire program. Finally, a certification system proves trainers’ and trainees’ understanding of fundamental topics of AI and Robotics [41, 49].

After developing the training and certification system (including a pilot implementation and evaluation), the first trainer and trainee trainings were implemented between the second quarter of 2019 and the first quarter of 2020. In sum, 32 courses were held with a total of 445 participants. A combination of qualitative and quantitative methods was used to evaluate the entire program. Participants deemed the blended learning approach as an essential factor for their learning success. This shows that the combination of hands-on focused face-to-face units and online units is the key to motivate people and to successfully impart complex AI/Robotics topics at the K-12 level. This could hardly be achieved by solely providing online sessions. The results also show that the main goals of the program (as described in Sect. 3.1) have been achieved for the most part. Course topics, content preparation as well as applied teaching methods were appropriate and trainers’ expectations towards the training were largely met. On the contrary, the evaluation also revealed the need for improvement regarding certain aspects of the program (as described in the previous section).

Due to the high demand, further trainings and certifications will be conducted in 2021 and in the years that follow. In order to get a more founded assertion regarding the success of the entire program, a more extensive quantitative evaluation will be implemented. This also includes revising and extending the certification system (revising questions, adding additional questions). Furthermore, contents and structure of the training modules are adapted according to insights, feedback and lessons-learned from the first implementations and evaluation results. This includes - among other things - revising AI and Python online teaching units and closing the gap between basic and advanced modules (as described in the previous section).

A deeper understanding of fundamental concepts of AI and Robotics enables people to make informed decisions and allows for a social and economic participation in the 21st century. The high demand of teachers wanting to participate in the EDLRIS trainer courses highlights the importance of sound trainings and certifications in these areas. By applying the innovative approach - a standardized and widely recognized training and certification system for teaching the principles of AI and Robotics at K-12 level (complex topics adapted and prepared for both beginners and more experienced people) - we envision to foster AI/Robotics literacy on a broad basis.