1 Introduction and background

Classroom response systems (CRS), also known as student response systems (SRS), or simply known as "clickers", represent an innovative educational technology designed to promote active learning and student engagement (Bruff, 2009). Used primarily in large lecture classes, these systems allow instructors to pose questions to students, who respond using a handheld device or their own mobile devices. Although these technologies were developed back in the sixties-seventies (Casanova, 1971; Judson & Sawada, 2002), they became more popular in the last decade thanks to digitalization (Gumaelius et al., 2023). Bring your own device (BYOD) concept allows easier accessibility since students can make use of their personal smartphone through different online platforms or applications (Afreen, 2014). The responses are immediately aggregated and displayed, providing real-time feedback on student understanding. This can help educators to adjust their instruction on the spot, addressing any misconceptions and promoting class discussion (Siau et al., 2006).

Moreover, one of the main advantages of using CRS is the promotion of student participation, as they allow even shy or hesitant students to voice their opinions anonymously (Altwijri et al., 2022). Across different studies it is proven that classroom response systems represent an effective intersection of technology and pedagogy, capable of enhancing both teaching and learning processes (Beatty et al., 2006; Fies & Marshall, 2006). Findings suggest that that the use of CRS in combination with different technologies and strategies (Caron & Gely, 2004) promote active learning classrooms. For instance, cluster-style seating can increase student engagement in large engineering classes when using CRS (Shekhar & Borrego, 2018). Therefore, the use of these systems promotes a higher level of students’ interaction and motivation, which may contribute to reduce the students’ resistance towards engagement in crowded classrooms (Aljaloud et al., 2015; Shekhar & Borrego, 2018). In addition, CRS may boost the integration of technology and pedagogy, providing a new paradigm of formative assessment (Shi & Hargis, 2023). Thus, these systems provide not only detailed information to evaluate individual performance, but also to assess the performance of a group (Şahin, 2020), which could be especially relevant in higher education such as engineering courses where overcrowded classrooms may impact the quality of education (Mora et al., 2012).

Gamification techniques can be implemented in CRS to create a different atmosphere in the classroom, promoting competitiveness in a fun learning environment (Krath et al., 2021). Gamification elements such as points, leaderboards, badges, and levels have a powerful effect on students (Saleem et al., 2022). Several studies have proven that a gamified CRS session is better perceived by students than standard electronic quizzes (Ismail et al., 2019) and provides a higher academic performance (López-Jiménez et al., 2021).

There is a variety of CRS software available such as Socrative, Mentimeter or Kahoot! (Leon & Peña, 2022). Socrative is known for its simplicity and strong formative assessment features, allowing teachers to test student understanding through quizzes and polls. Mentimeter excels in creating engaging presentations with real-time polling, Q&A sessions, and visualization of responses, which helps in promoting a dynamic learning environment. Kahoot!, on the other hand, stands out due to its gamified approach, which incorporates fun and competition into learning. This approach significantly boosts student participation and motivation, as it transforms traditional quizzes into exciting, fast-paced games. By combining educational content with game-like elements, Kahoot! encourages active learning and higher levels of student engagement compared to the more straightforward formats of Socrative and Mentimeter. This tool has succeeded in enabling a very fast evaluation of the concepts explained during the course, with a friendly environment that enhances the participation of the students (Lashari et al., 2023); see Fig. 1. Furthermore, Kahoot! includes in the evaluation not only the percentage of correct answers, but also the time spent to answer correctly. This can be an important feature when considering test results in a subject evaluation because, if a student copies a correct answer from his or her mate, the original answer will obtain a better score.

Fig. 1
figure 1

a Screenshots from question and (b) scoreboard of a test in Kahoot!

Full size image

Kahoot! is one of the most widely used CRS worldwide (Kahoot!, 2024a), being played by over 10 billion people from more than 200 countries since it was launched in 2013 (Hanoa, 2023). The platform was originally developed within the framework of a research project initiated at the Norwegian University of Science and Technology in 2006 (Wang et al., 2007). Currently, it is one of the preferred CRS platforms at university teaching level (Zhang & Yu, 2021). One of the main reasons of their popularity is the implementation of gamification techniques (Paciarotti et al., 2021; Shareef & Rauf, 2022), resulting in a game-based student response system (GSRS) where the classroom is temporality transformed into a game show, where the teacher is the host and students are the contenders (Wang, 2015).

Through a literature review including almost a hundred of quantitative and qualitative studies using Kahoot!, a positive effect has been proven not only on learning performance but also in classroom dynamics, attuites, and students’ anxiety (Wang & Tahir, 2020). In particular, best practices resulting from their use in engineering courses are confirmed, where the results showed that students became more active learners, and most of them were motivated to review the content of the course before the class and attend the class (Chernov et al., 2021). The literature shows other interesting studies of the application of Kahoot! on different engineering disciplines such as mechanical, computer or aeronautical engineering (Lopez Arteaga & Vinken, 2013; Garcia‐Lopez & Garcia‐Cabot, 2022; Albero & Ibáñez, 2018;). From students’ point of view, the use of Kahoot! improve classroom dynamics, engagement, motivation and learning beyond what would be expected from traditional teaching methods (Basuki & Hidayati, 2019; Licorish et al., 2018). Similar feedback is given by educators, who highlight the possibility to make learning funnier and to promote more creative processes while teaching (Batsila & Tsihouridis, 2018). Nevertheless, some teacher-centric must be avoided to not negatively affect students’ experience when using these systems in lecture lessons (Nielsen et al., 2013). This is precisely why structured methodologies of its application need to be studied and evaluated for an effective use in classroom.

In order to take advantage of the features of CRS previously addressed, the Heat Engines Group of the School of Industrial Engineering at Universidad Politécnica de Madrid decided to use this technology in their subjects in 2019. This technology has been used to enhance the participation of students anonymously in the lessons but, also, to promote the attendance to lectures. This has been done by allowing the students to obtain an extra punctuation in their final mark. Kahoot! was selected by the Group due to user-friendly environment and, especially, to the fact that the time spent to answer is considered in the evaluation. This is an additional indicator that may potentially correlate with the student performance in the final exam of the course.

Given the large amount of data collected during the last five academic years after the implementation of this technique, the benefits of CRS on students learning and performance can be assessed. However, there is no previous study that systematically demonstrated which variable obtained via CRS correlates better to the knowledge acquired by the students. This is precisely why the present study aims to fill this gap and conduct a systematic study with all the data collected during the horizon span not only during the CRS activities but also using the student performance at the end of the course.

Therefore, the research questions of this work are: what is the metric resulting from the evaluation of students via CRS that best correlates to the marks obtained by each student in the final exam? Is the time spent to correctly answer each question a parameter that should be considered in the evaluation? The goal of the proposed work is to evaluate the different key performance indicators based on the results of the CRS methodology for a large sample of students including different subjects. This assessment includes the analysis of the correlation between the selected key parameter indicators of the CRS (such as number of correct responses, number of responses or time of response) and other evaluations taken during the course, including theory and numerical problem exams. Hence this manuscript aims to provide a thorough analysis of the effectiveness of the CRS methodology applied in the students learning process and performance along five academic courses. The structure of the manuscript is as follows: after this first introductory section, the methodology is addressed in Section 2, including the sample of the study, the teaching and evaluation and the selection of key performance indicators; the results are presented in Section 3, including the distribution and correlation between KPIs, the temporary evolution and the correlation with theory and numerical problem exam results; Section 4 presents the feedback given by the students; and finally, Section 5 is devoted to the conclusions of the work.

2 Methodology

This section presents the methodology followed by the work. First, the sample of study is presented, including the number of students, the subjects and courses considered. Then, the teaching and evaluation approach is introduced. Finally, the selection of key performance indicators is given.

2.1 Sample of the study

CRS have been employed by the Heat Engines Group of Universidad Politécnica de Madrid (UPM) since the course 2018–19. More specifically, it has been used in two MEng Industrial Engineering (MUII) subjects (Heat Engines and Thermal Machines (MMT) -for non-thermal-specialized students- and Heat Engines and Thermal Machines II (MMTII) -for students that studied thermal engines during their BEng studies), one subject of BEng Industrial Technologies -GITI- (Thermal Engines, MT) and two subjects of BEng Energy -GIEn- (Volumetric Engines and Machines (MMV) and Technology of Turbomachinery (TTb)). The courses in which CRS have been implemented include 1200 students, where 1011 students have responded at least to one CRS question and have passed at least one exam, see Table 1. The ratio of international students is below 1% and, therefore, the sample of non-national students is not sufficiently large to analyse its effect. Student lists provided by the university have been used for this research. Unfortunately, these lists do not include information such as the genre, the age, or the number of courses where students have been engaged to this subject.

Table 1 Students included in the lists of the subjects of the sample, and, between brackets, number of students included in the study per subject and academic year
Full size table

One must note that the period evaluated in this work includes the COVID pandemic: in March 2020 the Spanish Government and the remaining lessons of the second semester took place online. Nevertheless, CRS questions still took place during lockdown. Similarly, both semesters of academic year 20/21 took place in a hybrid approach, where part of the students could attend physically for some lessons (not all), but all students could attend online to all lessons.

1S and 2S stand for courses from the first and second semesters, respectively.

2.2 Teaching and evaluation approach

All courses included in this study have a length of 14 weeks, with 2 or 3 h of lessons per week. The implementation of CRS in the UPM Heat Engines Group is based on the continuous learning approach: weekly, just at the start of the lesson, 3–4 questions related to the previous week’s lecture are posed by means of the CRS, where students answer to the questions by means of their personal mobiles. To the authors’ experience, this methodology involves the following advantages:

  1. 1.

    It encourages the students to read their notes taken from the previous lecture, enhancing the continuous learning approach.

  2. 2.

    Students are encouraged to arrive to the class on time, as the questions at posed at the beginning of the class.

  3. 3.

    It enables identifying concepts that have not been correctly assimilated by the students. If any question has a particularly poor result, doubts are solved before continuing with the new subject.

  4. 4.

    The marks obtained in the CRS along the course are used as an evaluation method, allowing to increase the mark obtained in other evaluation by a maximum of 10%. This mark increase percentage is based on the score obtained from Kahoot! reports, so that students that obtain a total score higher than 75% of the best total score among the students increase the subject mark by + 10%, whereas other students have an increase proportional to the ratio of their total score by the threshold.

CRS questions are multiple-choice questions, where 2 to 4 possible answers (mainly 4) are given and only one is correct. These questions are designed weekly by the professors of each subject and, as explained in Sect. 2.3, wrongly answered questions do not convey a negative mark.

Regarding the general subject evaluation of the students, it includes at least one theory test exam and problem exam.

The theory test exams consist of 20 questions with 2 to 4 possible answers, where only one answer is correct, i.e. the same typology of questions is used for the CRS tests and for the theory exams. In this case, wrong answers convey a negative mark, which is obtained statistically: -1 point for two-choices questions, -0.5 points for threr-choices questions and -1/3 for four-choices questions.

For many of the courses included in the study, the problem exam is also evaluated as a test, where 15 possible answers are given to the student. When this evaluation methodology is used for problem-solving test, at least 5–6 short numeric problems are included in each exam. As the number of numeric choices is very large (15 choices), no negative mark is given to wrong answers.

In addition, some subjects include in the evaluation methodology a written exam, laboratory session reports and other reports related to thermal engines. However, these typologies of evaluation are not included in the study, as their methodology is completely different to the CRS questions.

2.2.1 Key performance indicators (KPI) included in the study

The reports that can be obtained from Kahoot! Include the following items:

  • Total score: It is the evaluation of the tests. For each question, the student can obtain a maximum score of 1000 pt: if the student does not answer correctly to the question, it obtains 0 pt; otherwise, the score depends on the time used to answer as follows (Kahoot!, 2024b):

    $$Sc=1000\cdot \left(1-\frac{t}{2T}\right)$$
    (1)

    where \(t\) is the time used to respond by the student and \(T\) is the available time to respond the question. Figure 2 shows the correlation between the time to answer a question correctly and the score obtained in the question for an available time to answer of 20 s. One can observe that a correct answer obtains a minimum score of 500 pt.

  • Correct answers: Number of questions answered correctly.

  • Incorrect answers: Number of questions answered incorrectly.

Fig. 2
figure 2

Correlation between the time to answer a question correctly and the score obtained in the question

Full size image

As the total number of questions is different for each subject, each academic year and, even, each group of a course, results have been homogenized dividing the metrics by the total number of questions posed to the students or answered specifically by each student, depending on the metric. As previously seen, the performance of students answering CRS can be evaluated by the score obtained or by the correctness of answers. Therefore, two groups of KPIs have been used: those based on the number of correct answers and those based on the total score provided by Kahoot!:

  • Number-based KPIs.

    1. o

      Correct answers per question asked: It is the ratio between the number of correct answers and the number of questions asked.

    2. o

      Correct answers per question answered: It is the ratio between the number of correct answers and the number of questions answered by the student.

  • Score-based KPIs.

    1. o

      Mean score per question asked: It is the total score along the course divided by the number of questions asked to the students. If a student attends all lessons and answers correctly to all questions immediately, he/she will obtain 1000 pt in this variable.

    2. o

      Mean score per question answered: It is the total score along the course divided by the number of questions answered by the student. If a student answers correctly to all questions immediately, he/she will obtain 1000 pt in this variable, independently of the number of lessons he/she attends.

    3. o

      Mean score per question answered correctly: It is the total score along the course divided by the number of questions answered correctly by the student. If a student answers to all questions immediately, he/she will obtain 1000 pt in this variable, independently of the number of lessons he/she attends, and the percentage of questions answered correctly.

The ratio between the mean score per question answered correctly and per question answered is the percentage of questions correctly answered, whereas the ratio between the latter and the mean score per question asked is the percentage of questions answered by the students.

2.3 Analytical methods used to analyse the correlation between data

In order to evaluate the data obtained, the correlation between different sets of data is studied:

  • Correlation between the KPIs and the attendance ratio.

  • Correlation among different KPIs.

  • Correlation between KPIs and theory and problem exams.

The correlation between the different sets of data is analysed by means of two variables:

  • Coefficient of determination, R2: It represents the proportion of the variance in the dependent variable that is predictable from the independent variable. For example, an R2 value of 0.8 means that 80% of the variance in the dependent variable is predictable from the independent variable.

  • Root mean square error, RMSE: Residuals are a measure of how far from the regression line data points are. RMSE is a measure of how spread out these residuals are. In other words, it tells you how concentrated the data is around the line of best fit.

R2 is a variable that indicates fit quality, being a scale-free measurements that allow the comparison between different type of data fits. On the other hand, RMSE is more sensitive to large errors, and allows to identify correlations that may induce very large errors.

3 Results

The results are presented as follows: first, the distribution of key performance indicators within the study and the correlation between them is presented. Then, the evolution of the results during the five courses considered, which include Covid pandemic years, is shown. Finally, the correlation of the KPIs with the exam results is analysed. In addition to the analysis of the data to find the KPIs that best represent the students’ performance, a final sub-section is included to present the feedback from the students regarding CRS-based methodology.

3.1 Distribution and correlation between the key performance indicators

Figure 3 shows the histograms of these selected key performance indicators for the sample of the study.

Fig. 3
figure 3

Histograms of the key performance indicators and the mean tome to answer correctly

Full size image

One can observe that there is a relatively large number of students obtaining a mean score per number of questions lower than 200 pt with a mean value of correct answers per question asked below 0.15. These students are likely to correspond to students that attend only to a few lessons without continuity or to those that, attending to some of the lessons, they answer randomly. In this sense, it is interesting to observe Fig. 4, which depicts the correlation between the mean score per answer, the mean score per correct answer and the ration of correct answers by number of answers, with the attendance ratio, i.e. the ration of questions answered. Figure 4a and b show that students attending few lessons have a high standard deviation in the corresponding KPIs, which justifies the large number of students with mean total scores below 200 pt.

Fig. 4
figure 4

Correlation between selected KPIs and the attendance ratio

Full size image

The mean score per question answered, which considers the percentage of questions correctly answered and the time spent to do it, has mean values around 400 pt. One can observe that the top scores for total questions and asked to number of questions answered are very similar, which correspond to students that attend most lessons. This does not happen with students that correspond to results in orange and red boxes in Fig. 3a and b, where some of them attend most lessons and others to very few; as a result, the distribution of bars in the histogram for mean score per question answered is different than for mean score per question asked. A similar trend is observed when comparing Fig. 3d and e, with more students obtaining values of correct answers per question answered between 0.5 and 0.7 higher than for correct answers per question asked. This increase corresponds to students that do not attend all lessons.

Finally, the mean score per question answered correctly is the variable that has a narrower range of points. This variable gives an idea of the time spent to answer the question when the answer is correct. Most students obtain a mean value around 800 pt, which corresponds to a time of 8 s when the available time is 20 s. Students obtaining mean values below 650 pt (16 s) might correspond to students that copy the value answered by their classmates. It is very interesting to observe Fig. 4b, which shows that the mean score per question correctly answered is very similar for students that attend lessons than for those that don’t. That is to say, the distribution of time spent to answer is independent of the attendance level.As previously observed, Fig. 3 helps to identify students with low attendance ratios to the lectures and with very low ratios of correct answers, which probably corresponds to non-responsible answers. However, these students are not deleted from the data sets, as the main objective of the paper is to identify the metric that best correlates with the performance of students on the final exam, including all profiles of students present in the class.

Furthermore, Fig. 5 shows the correlation between the different KPIs of the study, which helps understanding the results of the present work.

Fig. 5
figure 5

Correlation between the study KPIs

Full size image

In Fig. 5a one can observe the correlation between the mean total score per question asked and the mean total score per answer. Students achieving the same value for both KPIs are identified in the green box and correspond to students attending to all lessons. These students achieve mean scores from 200 to 700 pt, which implies that many of the students that attend the lessons do not study weekly, as intended by the teaching and evaluation methodology.

When comparing these two KPIs with the mean total score per question correctly answered, i.e. the time spent time to answer correctly, Fig. 5b and c, it is possible to identify some students that obtain low values for the former KPIs and high values for the last one, red boxes. These students correspond to students answering very fast, but randomly, trying to take advantage of Kahoot! scoring system.

Finally, Fig. 5d and e compare the number-based KPIs with the corresponding score-based KPIs. One can observe that there is a very high correlation, especially for “per question asked” KPIs, with R2 close to 0.97. For low values of the scores and ratios, the standard deviation increases with these values, identified between blue lines. This implies that the coefficient of variation remains constant. However, this trend changes for the student achieving very high scores, for which the root mean square error starts decreasing with score, as identified between green lines. This implies that students answering correctly most questions have a lower variation in the time spent to answer.

3.2 Temporary evolution of the data

In order to analyse the results per course, the error bars of the three main variables and the percentage of questions answered and correctly answered (per number of questions asked) are depicted in Fig. 6. One should note that BEng subjects correspond to the second half of the academic year, whereas the MEng subjects are given during the first half. This is important to analyse the effect of the COVID-19 pandemic: during the second term of academic year 2019/20 and during the complete 2020/21 most of lessons were given online, identified in blue boxes. This influenced the percentage of questions answered, which has maximum values in this variable during the courses with online lessons; as students were confined in their houses, it was easy for them to connect at the beginning on the lessons, answer the CRS questions. When face-to-face lessons came back, red boxes, there was a decrease of students attending the lessons, which leads to minimum values of questions answered. It is interesting to note that, for all subjects where the methodology was applied during the COVID pandemic, minimum values of mean scores per question asked and per question answered take place during the year after lockdown (and during the second of only lessons the second-semester subjects). In the authors’ opinion, it is surprising that when students could come back to university both the attendance and the performance of the students decreased, not only compared to the pre-pandemic, but also to the pandemic courses.

Fig. 6
figure 6

Mean values and standard deviations for (left) the percentage of questions answered and correctly answered and (right)

Full size image

Regarding the scores, one can note that the mean score per questions correctly answered maintains approximately constant for all courses (around 800 pt), which implies that the time spent to answer questions does not vary among subjects and course. Regarding the mean scores per questions asked and per questions answered, they vary similarly to the percentage of questions answered. This implies that the ratio of questions answered correctly per questions answered maintains constant for different subjects and years, which is coherent with the relatively constant mean score per question correctly answered.

Finally, orange boxes identify subjects of the Bachelor in Energy Engineering. The acceptance in this Bachelor Degree is less competitive than for the Bachelor Degree in Industrial Engineering and, therefore, the mean marks of the students accepted in the BEng Energy during high school studies are lower. Furthermore, the first two courses of this BEng belong to the School of Mining and Energy Engineering, whereas the third and fourth courses belong to the School of Industrial Engineering, where the present study has been developed. This implies that some of the students in the third course, where the MMV and TTb subjects take place, still must attend the School of Mining and Energy Engineering when they have some subjects that have not been passed yet. Both facts might explain why the students of these subjects have lower attendance ratios and poorer results.

3.3 Correlation between theory and problem exams

In this section, the correlation between both exams is analysed. Figure 7 presents the results of all students of the sample in their theory and problem exams (when more than one exam is passed along the academic year, the mean value of the different exams is used).

Fig. 7
figure 7

Correlation between the theory and numeric problem exams for the students of the sample

Full size image

One can observe that the results of both kind of test exams, theory and numeric problems, correlate fine with a linear function (R2 is similar using a quadratic or cube polynomial correlation). As the theory and numeric problem exams evaluate different student skills, this high correlation can only be explained by the fact that students who prepare very well for their exams acquire the competencies required for both kind of exams. Nevertheless, there are students that, obtaining a mean theory exam result around 4, obtain a mean numeric problem result from 0 to 10.

3.4 Correlation between the KPIs and theory exams

This section is devoted to the analysis of the correlation of the selected CRS variables with the theory exams. Figure 8 shows the correlation between the mean theory exam results and the selected key performance indicators for all students of the sample.

Fig. 8
figure 8

Correlation between the mean theory exam result and the CRS key performance indicators

Full size image

First, the KPIs that best correlate with the theory exam results of the students are sought. One can observe that R2 values for the total mean total score per question correctly answered is very low, 0.01. This implies that this variable is not adequate to be used in the continuous evaluation mark, which allows an increase in the final mark up to 10%. The low correlation between these variables is coherent with the inexistent correlation between the KPI and the attendance ration, see Fig. 4b. On the contrary, the variables that best correlate to the mean theory exam are the ones based in “per question asked” values, Fig. 8a and d, with R2 values around 0.21 compared to 0.13–0.14 for the “per question answered” KPIs.

Finally, number-based and score-based KPIs are assessed. When comparing Fig. 8a with d, and Fig. 8b with e, it results that the difference in the correlation based on the R2 (0.206 vs 0.209 and 0.134 vs. 0.138) and on the non-dimensional value of the root mean square error (0.140 vs 0.170 and 0.113 vs. 0.130) are very low: the correlation is very slightly higher for the number-based KPIs, but root mean square errors are also higher. These results do not allow to conclude that the use of one or the other KPI is more recommended.

As it has demonstrated in Fig. 8 that the KPIs that best correlate with the exams are clearly those based in the number of question asked, and not in the number of questions answered or correctly answers, Fig. 9 only represents the correlation between these two KPIs and the results of the numeric problem exams.

Fig. 9
figure 9

Correlation between the mean numeric problem exam result and the CRS key performance indicators based in number of questions asked

Full size image

Comparing Figs. 7 and 8 it results that the correlation of the KPIs with the numeric problem exams are notably worse than the correlation with theory exams, with R2 of 0.15 compared to 0.21. This is consistent with the fact that the questions asked during the course by means of classroom response systems are very similar to those asked in the theory exams, whereas the competencies required to solve problems are different. However, it can be assumed that the still existing correlation between the numeric exams is because students that attend all lessons and study the subject daily normally prepare very well for their exams.

Surprisingly, when analysing the correlation with numeric problem results there is a higher R2 for total score per question asked than for number of correct answers per question asked, 0.155 compared to 0.148, being non-dimensional value of the root mean square error notably lower, 0.149 vs 0.181. The authors have not found a reason that justifies this result when, at the same time, R2 is the same for both KPIs with theory exams.

4 Student feedback

As in most universities, at the end of the course student might complete an anonym survey. This survey system depends on the University and, thus, there is no specific question related to the CRS. Nevertheless, student might include free comments in the survey and, normally, positive comments are written for the evaluating the CRS. The following are some of the comments received:

  • I like playing Kahoot! at the beginning of classes.

  • The Kahoot! tests each week are a good way to stay a little more up-to-date.

  • The Kahoot! tests are very helpful; they help me stay on top of the subject.

  • I like following the lessons with Kahoot; it forces me to understand and study, keeping up-to-date.

  • The small evaluations before each class require staying current with the subject.

  • The Kahoot! tests give you an idea of the exam.

  • By doing a Kahoot! test every day at the beginning of class on the lesson learned, the concepts were assimilated more easily.

  • Doing a Kahoot! helps you stay on top of the subject and shows more interest in it.

There were only a few negative comments related to the use of Kahoot!, but they always addressed the fact that the test takes place at the very beginning of the lessons and, thus, student arriving late cannot be evaluated.

5 Conclusions and future works

This study has analysed the correlation between the classroom response system results and the exam results across five academic years including more than a thousand students of Universidad Politécnica de Madrid. The following conclusions can be extracted from the work:

  • The KPIs that best correlate with the results obtained by the students in the exams are those based on the number of questions asked, i.e. students that attend more frequently to the lessons obtain better marks in their exams and, thus, KPIs based on the number of answers or the number of correct answers should be discarded.

  • The correlation with theory exams is better than for numerical problems exams. This is consistent with the

  • It cannot be stated that the use of the time spent to answer the question has an advantage compared to the use of percentage of correct answers. Therefore, the selection of a given CRS technology should not be based on the fact that it considers the time to answer correctly to evaluate the answer.

  • The feedback of the students to the methodology used is very good, as they realize its usefulness to keep up-to-date the subject and to prepare for the exams.

These conclusions will be especially helpful for those teachers that use CRS not only for the sake of student engagement thanks to gamification methods, but also for partially evaluating the knowledge acquired by the students. All CRS available in the market can feed the teacher with the number of correct answers of each student but, to the authors knowledge, only one gives a score based on the time spent by each student to answer correctly. The main conclusion of this work is that using this metric does not involve an advantage compared to the number of correct answers and, thus, the selection of the CRS can be done based on other reasons.

This work has been developed with students from different degrees, but all of them are related to engineering and in the same centre. A multi-centre study, including centres from different countries and technological and non-technological degrees would increase the generalization of the study. Information such as the sex, genre and age of the students has not been included in the research, as it was not available. Future works might include this information to enlighten about its effect. Nevertheless, authors believe that the main finding of the study, i.e. time to respond correctly is not a variable that improves the evaluation of the students, will not depend on the centre or the country.

On the other hand, the main limitation of this study is that most students are aware of how the score of Kahoot! is obtained. Therefore, students that do not study the subject weekly might take the decision to answer very fast randomly, as this would have a benefit in their score compared to trying to understand the question. If other CRS is used, only considering the number of correct answers, the behaviour of these students might change, e.g. students that do not study weekly might prefer to wait and try to copy the answer of their classmates.