Keywords

1 Introduction

Educational Robotics (ER) is based on the constructionist learning theory proposed by Papert (1980). According to this theory, the construction of a personal and meaningful artifact promotes the construction of deep knowledge and provides a richer learning experience for students. When students design, build, program, debug and share robots, they are exploring the everyday phenomena in their lives in a new and playful way (Resnick et al. 1996). They are also improving their problem-solving and thinking skills (Scaradozzi et al. 2015, 2019b).

The spread of ER projects in schools all over the world (Miller and Nourbakhsh 2016) and the growing interest of teachers and policy-makers in this approach justify the research into new and more far-reaching ways to evaluate students’ results. Researchers in the ER field have found a lack of quantitative analysis on how robotics can improve skills and increase the learning achievements of students (Benitti 2012; Alimisis 2013). For this reason, studies in recent years have focused on the evaluation of ER projects, employing qualitative (Liu 2010; Elkin et al. 2014), quantitative (Kandlhofer and Steinbauer 2016; Cesaretti et al. 2017; Scaradozzi et al. 2019a) and mixed methods (Kim et al. 2015; Chalmers 2018) to collect educational data. During a typical ER activity, students have to design a hardware or software solution to an open-ended problem and, generally, there are unlimited paths students can explore and implement. Using machine learning methods to collect and analyze ER data could help predict and classify students’ behaviors, and discover latent structural patterns in large educational datasets (Berland et al. 2014).

Most studies in recent years (Berland et al. 2013; Blikstein et al. 2014; Chao 2016; Wang et al. 2017) have applied machine learning techniques to data gathered from students during programming activities without the physical presence of robots, and have obtained good results in identifying different patterns in specific coding tasks. This paper presents a research project that aims to analyze an introductory ER exercise using educational data mining techniques. The authors designed a tracking system to register log files containing the programming sequences created by (teams of) students in several secondary school classes. In this preliminary study, the collected data were used as inputs for machine learning algorithms, with two main objectives: 1—to identify different patterns in the students’ problem-solving trajectories; 2—to accurately predict the final performance of the teams of students.

2 Methods

2.1 Procedure and Participants

Our experiment takes the existing research further by using educational robots. After upgrading the Lego Mindstorms EV3 programming blocks—using the Lego Mindstorms Developer kit (2019)—the authors were able to register several log files containing the programming sequences created by 197 Italian secondary school students (organized into 56 teams), while solving an introductory Robotics exercise. Using the data collected as input for machine learning algorithms, the authors tried to identify different patterns in the students’ problem-solving trajectories and predict the final performance of their teams. The experiment was conducted from March 2018 to March 2019, and involved seven Italian lower/higher secondary schools (located in the Emilia Romagna and Marche regions). The authors used convenience sampling to involve students in the experiment. Data were collected from the robotics courses organized in the schools by the startup TALENT (a partner in the research project). Two of the schools decided to offer a curricular robotics class, five decided to offer a non-curricular robotics class. 18.78% of the participants had already used robotic kits at home or at school before the experiment.

2.2 The Introductory Exercise

One of the first steps in the “Introduction to Robotics” course, whose aim was to get students involved in the experiment, was an exercise with the Lego Mindstorms EV3 motors. They had to program the robot to cover a given distance (1 m), while trying to be as precise as possible. The educator suggested a specific block to turn on the robot’s motors. Using the “Move Steering” block, students can select one of the four different options shown in Fig. 1 (On for Rotations, On for Degrees, On for Seconds, On). By modifying the numeric parameter highlighted in Fig. 1, the students’ team can specify how many rotations the wheels have to complete. Following the same strategy, students can decide whether to define how many degrees or for how many seconds the motors should rotate. The first two numeric parameters are for steering the robot and the speed of the motor.

Fig. 1
figure 1

Lego Mindstorms EV3 “Move Steering” block

Full size image

To solve this exercise, the teams had to design their solution in a maximum amount of time (15 min for upper secondary school classes, 20 min for lower secondary school classes), using no measuring tools (set squares, rulers, etc.) to check the distance covered by the robot on the floor. The measurements only applied to some of the robot’s parameters (e.g., the wheel radius). The instructor gave a final evaluation for this exercise based on how accurate the team was. If the error was ≤4 cm, the educator considered the challenge completed; if the error was ≥4 cm the educator considered the challenge not completed.

2.3 Data Preparation

The tracking system collected 1113 programming sequences created by the teams of students to solve the robotics exercise described in the previous section. Several technical steps had to be performed to transform these sequences into matrices. After this transformation, the following nine indicators were calculated for each programming sequence (using a function in Python that parses the log file): Motors (how many motor blocks are in the sequence); Others (how many blocks in the sequence belong to categories other than Motors); Added (how many blocks have been added, compared to the previous sequence); Deleted (how many blocks have been deleted, compared to the previous sequence); Changed (how many blocks have been changed, compared to the previous sequence); Equal (how many blocks have stayed the same, compared to the previous sequence); Delta Motors (the degree of change in Motor block parameters, compared to the previous sequence, and calculated only for blocks in the “Changed” category); Delta Others (the degree of change in Other block parameters, compared to the previous sequence); Trials (how many times students activated the programming sequence on the robot).

3 Results

The authors adopted two machine learning approaches, with the aim of predicting the teams’ results:

  • A supervised approach, whereby the mean value and the standard deviation were calculated for each indicator presented in the previous section, and were used to create a feature matrix as the input for four machine learning algorithms (logistic regression, support-vector machine (SVM), k-nearest neighbors, and random forest classifier); the authors compared how these algorithms performed, predicting the final results of the teams of students;

  • A mixed approach, which combined the benefits of both the supervised and the unsupervised methods: a k-means algorithm was applied to calculate what clusters might form from the programming sequences. Then the log files were analyzed to calculate the percentage of sequences belonging to each cluster: these new features were then used to create a feature matrix as input for the four supervised algorithms mentioned previously.

A repeated tenfold cross validation was performed (Kim 2009) to evaluate the effectiveness of the different strategies at predicting the educational results, and four parameters were calculated (Huang and Ling 2005): Accuracy, Mean Precision, Mean Recall, Mean F1—Score. The best performance was obtained with the SVM algorithms combined with the features calculated by the k-means algorithm (the mixed approach). This combination led to the following predictions of the students’ performance at resolving the exercise (challenge completed or not completed): Accuracy = 0.89 (SD = 0.002), Mean Precision 0.84 (SD = 0.007), Mean Recall = 0.84 (SD = 0.011), Mean F1 Score 0.83 (SD = 0.011).

Three problem-solving strategies emerged from the dataset collected. The “Mathematical strategy” groups performed a low number of total tests (compared to the other strategies), all of them with a rotation parameter equal to a fixed value (see Fig. 2 in Appendix, which shows the trend of the rotation parameter set by a group of students taken as an example for this approach). They probably used a mathematical formula to choose the robot’s parameters, and never changed the value in the Motor block. The “Incremental Tinkering approach” groups performed a medium number of tests (compared to the other strategies); they probably used a heuristic approach to choose the robot’s parameters, refined them repeatedly (see Fig. 2 in Appendix) and analyzed the feedback from the robot (for a definition of tinkering see Resnick and Rosenbaum (2013)). The “Irregular Tinkering approach” groups performed a high number of tests (compared to the other approaches), and their strategy is characterized by a broken line representing the number of seconds set in the Move Steering block (see Fig. 3 in Appendix), and a broken line representing the Delta Motors value calculated between two contiguous tests (which not only included changes to the number of seconds, but also modifications to the speed and steering parameters, (see Fig. 4 in Appendix). This last group probably also used a heuristic approach to choose the robot’s parameters, but a certain number of large modifications could indicate difficulty understanding the feedback from the robot. These three behaviors were the most common in the dataset analyzed.

4 Conclusions

This experiment presented a number of innovative features. To the best of the authors’ knowledge, this is the first experiment involving a fair number of students (197) to gather and look at programming sequences designed by students during ER activities, and analyze them using machine learning techniques. Good results were obtained predicting the students’ performance, especially in terms of Accuracy (0.89). The three styles presented in the Results section are close to that of the “planner scientist” (similar to the “Mathematical strategy”) and the “bricoleur scientist” (similar to the “Incremental Tinkering approach” and to the “Irregular Tinkering approach”) proposed by Turkle and Papert (1992). The promising results of this preliminary study have encouraged the authors to involve new groups of students in the experiment, and also to collect data for different types of challenges, in order to validate the approach.