1 Literature review

Over the past decade, medical and health care education have emphasized the importance of active learning and technology to improve students’ engagement and critical thinking skills (Freeman et al., 2014; Stewart et al., 2011). Active learning represents a shift away from a traditional lecture that tends to render learners bored or passive (McLaughlin & Mandin, 2001; Wolff et al., 2015). This method emphasizes higher-order thinking, often involves group work (Freeman et al., 2014), and is effective in improving learning outcomes in medical education (Graffam, 2007; Michael, 2006).

Large class teaching is known to be challenging for implementation of active learning strategies. Audience response system (ARS), defined as an interactive electronic tool used to survey audiences’ responses to specific questions (DeSorbo et al., 2013), is one of technology tools that can be applied to enhance active-learning environment (Allen & Tanner, 2005) and participation (Bright et al., 2013) especially in a large class. ARS can develop a didactic lecture into an interactive learning process (Miller et al., 2003); as it encourages each student to take an active part in the discussion in various levels with anonymity (Cain & Robinson, 2008), addresses misinterpreted content, and facilitates adequate and relevant feedback from the teachers (Acharya, 2001; Gwee & Hoon, 2001).

Kahoot, a simple-to-use online tool, allows educators to create gamified, customized interactive quizzes and surveys; can be accessed by students on their own mobile devices or computers (Bryant et al., 2018); and is designed to make learning more fun and engaging with audio-visual stimuli, music accompaniment, real-time results, and immediate feedback (Ismail et al., 2019). Kahoot is widely used in universities for classroom activities, review sessions, and formative assessment (Wang & Tahir, 2020). Even though it is available for free, there are many limitations, including number of participants, availability of question formats, accessibility of assessment reports, and feature customization.

To play Kahoot, students log into a game using a PIN code or a link, participate anonymously with the name of their preferences or class-rule regulations, and then answer multiple-choice or true/false questions or other types of questions by themselves (Lohitharajah & Youhasan, 2022). Players compete against each other for high scores in a time manner (Brown et al., 2018). The scores are based on the answering speed and the number of questions answered correctly (Lohitharajah & Youhasan, 2022), and can be categorized as Kahoot scores and corrected scores. Kahoot scores are points earned by students for both the correctness and speed of their responses to questions, while corrected scores specifically refer to the points earned for answering a question correctly.

Kahoot can be categorized into two types: Kahoot for individual play (Kahoot-I) and team play (Kahoot-T). Kahoot-I offers students a personalized learning experience that enhances motivation, engagement, self-directed learning, personalised feedback (Ismail et al., 2019), and anxiety reduction (Castillo-Manzano et al., 2016); however, it may limit social interaction and data response collection for further analysis. Notably, Kahoot-T allows players to collaborate as a team leading to enhanced teamwork and communication. However, as its time constraints; limited interaction between team members, limited team strategy, and potential for group dominance should be considered.

Google Forms is an online survey platform that allows for the creation of various question types, such as multiple-choice questions (MCQ) and open-ended questions (OEQ) (Google Forms-Create and Analyze Surveys, for Free). In addition, Google Sheets is a platform that stores, processes, and analyses data based on user-defined formulas and can create data visualisation in many platforms, such as pie charts and bar graphs (Google Sheets-Overview and Features). Both Google Forms and Google Sheets are free to use and have no limitation on the number of participants (Haddad & Kalaani, 2014).

2 Introduction

To enhance the use of both Google Forms and Google Sheets, we combined them into two feasible ARS formats, Google-MCQ and Google-OEQ, to serve question types designed by the instructors to meet learning purposes of the class. These formats allow students to respond to MCQ and OEQ questions via their own devices and then the answers are analysed and displayed on the screen. This approach helps overcome the limitations of Google Forms alone and promotes better data analysis and presentation. However, learning curve and expertise are required to maximize the benefits of these platforms.

Google-MCQ has several advantages, including user-friendly format, modifiable and flexible number of choices, self-paced responses without time limitation, and instantaneous grading and feedback after submission (Haddad & Kalaani, 2014; Nguyen, 2018). However, the absence of immediate feedback while performing each task and the unappealing interface may hinder students’ engagement.

Google-OEQ offers several advantages, most notably allowing students to freely express their opinions without limitation from providing a list of potential responses (Atilgan et al., 2020; Gharehbagh et al., 2022). This can promote in-depth and detailed responses, critical thinking of students, accurate and honest feedback, and reflection opportunities (Gharehbagh et al., 2022; Husain et al., 2012). Nevertheless, analysing OEQ responses can be challenging, time-consuming, and subjective, which may delay scoring and feedback (Atilgan et al., 2020; Gharehbagh et al., 2022). In large classes, Kahoot, Google Forms, and Google Sheets are made available without charge by our faculty. Initially, we were uncertain whether these tools could effectively engage our students. Furthermore, we lacked information on which Audience Response System (ARS) aligns with students’ preferences for different types of activities. The rationale for comparing the effects of Kahoot and Google Forms and Sheets on students’ learning objectives, academic outcomes, and preferences was to align the fundamentals of the classroom, including learning objectives, determining evidence of learning (academic outcomes), and enhancing learning experiences (engagement and preferences) (Wiggins & McTighe, 2005). Previous literature suggests that some learning methods, although not always satisfying for learners, can benefit their academic outcomes (Deslauriers et al., 2019). This approach aimed to find which ARS is the most suitable tool to optimize student learning objectives, impact academic outcomes, and align with student preferences. Such an approach ensures that the chosen ARS is both educationally effective and well-received by the students, thereby facilitating a more engaging and satisfying learning environment.

To achieve our aims, we designed questions in the questionnaire based on student engagement dimensions (Kassab et al., 2023) and divided them into aspects that are easily ratable and understandable by students. Student engagement encompasses five dimensions: behavioral, emotional, cognitive, agentic, and socio-cultural (Kassab et al., 2023). Our focus was on the first four dimensions, including “participation”, “attention”, friendly peer competition (“competition”) for the behavioral dimension; “enthusiasm”, “liking”, “happiness”, feeling interested in class (“interested”) for the emotional dimension; “understanding”, capturing concept (“concept”), following content (“following”) for the cognitive dimension; and courage to ask and answer questions (“asking/answering”) for the agentic dimension.

It is important to critically consider the advantages and limitations of each ARS format to ensure they align with learning objectives, academic outcomes, and student preferences. In our faculty, which has over 300 students in each class, the choice of ARS formats may be limited due to the regulations of each ARS format and budget constraints. Since our students enter medical school directly after high school graduation, which may impact their level of maturity, they might not be highly engaged or internally motivated in large class environments (Luscombe & Montgomery, 2016; White et al., 2014). This study aimed to (1) evaluate students’ perspectives regarding benefits of each ARS in various aspects, including augmenting students’ “participation”, “attention”, “understanding”, “enthusiasm”, “happiness”, “liking”, “competition”, “asking/answering”, “concept”, “interested”, and “following” with or without subgroup analysis of students into quartiles of the summative score, (2) compare students’ perspectives on each aspect in many ARS formats with or without subgroup analysis of students into quartiles of the summative score, and (3) determine correlations between students’ perspectives and academic outcomes. Revelation of students’ perspectives and academic outcomes on using different ARS formats might lead to choosing the appropriate tool(s) for enhancing students’ learning.

3 Materials and methods

3.1 Study protocol and data collection

The study protocol was approved by the Siriraj Institutional Review Board (Certificate of Approval no. Si 181/2019) on March 14th, 2019. The subjects of this study were medical students from the Doctor of Medicine program, Faculty of Medicine Siriraj Hospital, Mahidol University, Thailand which consists of six years, including premedical year (Year 1), the 1st preclinical year (Year 2), the 2nd preclinical year (Year 3), and clinical years (Year 4–6). Participants were a cohort of the medical students of class of 2021 who were enrolled in the “SIID 325: Disorders of endocrine and multi-organ systems” subject, taught in the 2nd preclinical year during November, 2018 to March, 2019.

Different formats of ARS, including Kahoot-I, Kahoot-T, Google-MCQ, and Google-OEQ, were employed in five large classes (312 students/class) in the SIID 325 subject to match with different types of activities in each class. Our institution provides Kahoot for free and can accommodate over 400 participants. Additionally, Google Forms and Sheets are available for free with unlimited participants, meeting our requirements.

Kahoot-I was used in a class that aimed to promote enjoyment, excitement, and engagement with self-assessment (Ismail et al., 2019). Kahoot-T was used in a class with similar objectives to Kahoot-I but encourages students to have more discussions with friends in a team. Google-MCQ was used in a class that required assessing students’ knowledge with their self-paced progress before and after class, as the instructor could estimate the students’ understanding and feedback the answer immediately (Haddad & Kalaani, 2014; Nguyen, 2018). Google-OEQ was implemented in a class that aimed to (1) encourage and induce students to answer questions with higher thinking processes to articulate creativity (Atilgan et al., 2020; Gharehbagh et al., 2022), (2) allow students to freely, comfortably, and instantaneously ask questions, and (3) and provide opportunities for students to express their opinions and feedback with anonymity (Gharehbagh et al., 2022; Husain et al., 2012). In some classes, more than one ARS formats were used to optimize the learning outcomes. The ARS format(s) selection diagram illustrating how ARS format(s) was/were chosen based on instructional objectives is shown in Fig. 1.

Fig. 1
figure 1

The ARS format(s) selection diagram

Full size image

MCQ: Multiple-choice question, OEQ: Open-ended question, Kahoot-I: Kahoot for individual play, Kahoot-T: Kahoot for team play, Google-MCQ: Google Forms and Google Sheets with multiple-choice question, Google-OEQ: Google Form and Google Sheets with open-ended question.

Kahoot-I, Google-MCQ, and Google-OEQ were used to ask questions before, during, and after associated content provided by the instructors. Google-OEQ was used as a tool in every class letting students anonymously ask questions and freely provide their opinions and feedback. In the formative assessment, after using Google-MCQ to test individual’s knowledge, Kahoot-T was also used to enable students to discuss and answer the questions in the team mode. In addition, in some group activities, Google-MCQ was used for a specific objective in which pretest questions were similar to posttest questions (“SimilarPrePost”) to evaluate students’ progress. For “SimilarPrePost”, questions regarding main concepts of the class were asked by Google-OEQ at the beginning of the class and the same questions were asked again with the same format at the end of the class.

In Class 3, Kahoot-I was used three times with asking students to use their student ID as username. In the first two Kahoot sessions, students answered recall questions while the third session involved critical thinking questions. The selection of ARS format(s) according to instructional objectives is(are) shown in Table 1.

Table 1 The selection of Audience Response System format(s) according to instructional objectives
Full size table

3.2 Questionnaire

At the end of the final class of the SIID 325 subject, a QR code for the Thai-language questionnaire was displayed on a screen. We constructed our questionnaires based on students’ engagement frameworks (Kassab et al., 2023) and in consultation with field experts, as the questionnaires in the existing literature (Kassab, Al-Eraky, Kassab et al., 2023a, b). The questionnaire was created by Google Forms which consisted of 3 main parts. The first part asked students’ perspectives in using different ARS formats in various aspects compared to traditional lectures which are shown in Table 2. Students’ perspectives were rated using a Likert scale as followings: 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree.

Table 2 The questionnaire asking students’ perspectives in using different ARS formats in various aspects compared to traditional lectures
Full size table

The second part asked students to choose the format(s) of ARS that could promote their learning in each aspect. The last part was open-ended questions asking students to provide their positive and negative comments regarding using ARS in class.

The clarity and legibility of the questionnaire were initially reviewed by staff members in the physiology department. Subsequently, the questionnaire was submitted to the expert committee for validation, on rational analysis, comprehensiveness, readability, and clarity of the questionnaire. The internal consistency of the data, calculated with the Cronbach’s alpha formula, was 0.838.

Medical students who voluntarily returned written informed consents and completed the research questionnaire were recruited in this study. There were 86.22% (269/312) respondents with 54.6% (147) males and 45.4% (122) females. The mean ± standard error of the mean (S.E.M.) age was 20.46 ± 1.44.

3.3 Academic achievement

Academic achievement, represented as the SIID 325 formative and summative scores, was obtained from the undergraduate education department.

3.4 Subgroup analysis

Students were allocated into 4 groups based on the quartiles of the SIID 325 summative scores which Q1 and Q4 represented groups of students with the lowest and highest score, respectively.

3.5 Statistical analysis

Data analysis was performed by the Statistical Package for Social Science version 18. Descriptive statistics were used to analyze number of students who choose the format(s) of ARS that could promote their learning in each aspect. Non-parametric tests were used to compare rank data between groups of different ARS formats or different aspects of students’ perspectives or quartiles of the summative score. The correlation coefficient (R value) was used to represent correlations between 2 factors, which was performed by the Spearman’s rank-order correlation coefficient. Multiple linear regression was used to test which factors significantly contributed to formative scores, summative scores, and the percentile of summative scores. A statistically significant difference was set at P value less than 0.05.

4 Results

4.1 Comparisons of students’ perspectives between various aspects in each ARS format compared to traditional lectures

Comparisons of students’ perspectives between various aspects in each ARS format compared to traditional lectures are shown in Fig. 2. From asking students’ agreement whether ARS could promote their learning, students rated the highest scores for “participation” for Kahoot-I (Fig. 2A), Kahoot-T (Fig. 2B), Google-MCQ (Fig. 2C), Google-OEQ (Fig. 2D), and overall ARS (Fig. 2E). Scores for “understanding” were rated lowest for Kahoot-I (Fig. 2A) and Kahoot-T (Fig. 2B); scores for “liking” were rated lowest Google-MCQ (Fig. 2C) and Google-OEQ (Fig. 2D); and scores for “happiness” were rated lowest for overall ARS (Fig. 2E).

For the particular objective, “SimilarPrePost”, students rated the highest scores for “liking”, “understanding” and “interested”; followed by “concept”, “enthusiasm”, “attention”, “following”, and “happiness”, respectively (Fig. 2F).

Fig. 2
figure 2

Comparisons of students’ perspectives between different aspects in each ARS format compared to traditional lectures. aP < 0.05, aaP < 0.01, aaaP < 0.001 compared to “participation”, bP < 0.05, bbbP < 0.001 compared to “attention”, cP < 0.05, ccP < 0.01, cccP < 0.001 compared to “understanding”, dP < 0.05, ddP < 0.01, dddP < 0.001 compared to “enthusiasm”, eeP <0 .01, eeeP < 0.001 compared to “happiness”, fP < 0.05, ffP < 0.01 compared to ‘liking” or “competition” or “asking/answering”, gP < 0.05 compared to “concept”, 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree

Full size image

4.2 Comparisons of students’ perspectives between different formats of ARS in each aspect compared to traditional lectures

Comparisons of students’ perspectives between different formats of ARS in each aspect compared to traditional lectures are shown in Fig. 3. When using ARS in the particular objective, “SimilarPrePost” was rated highest when compared to other types of ARS in various aspects, including “attention” (Fig. 3B), “understanding” (Fig. 3C), “enthusiasm” (Fig. 3D), “happiness” (Fig. 3E), and “liking” (Fig. 3F) (P < 0.01 all).

When the comparisons were made between different formats of ARS without the particular objective, “SimilarPrePost”, scores for Kahoot-T were rated highest among different ARS formats in “participation” (Fig. 3A), “attention” (Fig. 3B), “enthusiasm” (Fig. 3D), “happiness” (Fig. 3E), “liking” (Fig. 3F), and “competition” (Fig. 3G).

For “understanding”, students rated the highest scores for Google-MCQ, followed by Google-OEQ, Kahoot-T, and Kahoot-I, respectively (Fig. 3C).

Fig. 3
figure 3

Comparisons of students’ perspectives between different formats of ARS in each aspect compared to traditional lectures. aP<0.05, aaP<0.01, aaaP<0.001 compared to Kahoot-I, bP<0.05, bbP<0.01, bbbP<0.001 compared to Kahoot-T, ccP<0.01, cccP<0.001 compared to Google-MCQ, dddP<0.001 compared to Google-OEQ, 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree

Full size image

4.3 Comparisons of students’ perspectives between various aspects in each ARS format compared to traditional lectures with subgroup analysis of quartiles of the summative score

Comparisons of students’ perspectives between different aspects in each ARS format compared to traditional lectures with subgroup analysis of quartiles of the summative scores are shown in Fig. 4. For Kahoot-I, scores for ‘enthusiasm’ were rated significantly higher in the Q4 students compared to the Q1 and Q2 students (P < 0.05 all) while scores for other aspects were comparable between groups (Fig. 4A).

For Kahoot-T, scores for “understanding” were rated higher in the Q2 and Q3 students compared to the Q1 students and scores for “happiness” and “liking” were rated higher in the Q2, Q3, and Q4 students compared to the Q1 students (P < 0.05 all) (Fig. 4B). Scores for all aspects in Google-MCQ (Fig. 4C), Google-OEQ (Fig. 4D), overall ARS (Fig. 4E), and “SimilarPrePost” (Fig. 4F) showed no significant difference between groups of quartiles of the summative score.

Fig. 4
figure 4

Comparisons of students’ perspectives between different aspects in each ARS format compared to traditional lectures with subgroup analysis of quartiles of the summative score. aP<0.05, aaP<0.01 compared to Q1, bP<0.05 compared to Q2, 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree

Full size image

4.4 Students’ perspectives of type(s) of ARS that could promote their learning in each aspect

The percentage of students who voted the type(s) of ARS that could promote their learning in each aspect and the type of ARS that they liked most are shown in Fig. 5. For all aspects, there were around 60 to 85% of students voting for Kahoot-T, around 50–60% of students voting for Kahoot-I and Google-MCQ, and only around 20% of students voting for Google-OEQ in promoting their learning (Fig. 5). For the type of ARS that students liked the most, 62.9% of them voted for Kahoot-T, while 19.4%, 15.1%, and 2.7% of them voted for Google-MCQ, Kahoot-I, and Google-OEQ, respectively (Fig. 5).

Data are shown as the percentage of students who voted the type(s) of ARS that could promote their learning in each aspect and the type of ARS that they liked most.

Fig. 5
figure 5

Students’ perspectives of type(s) of ARS that could promote their learning in each aspect. Kahoot-I: Kahoot for individual play, Kahoot-T: Kahoot for team play, Google-MCQ: Google Forms and Google Sheets with multiple-choice question, Google-OEQ: Google Form and Google Sheets with open-ended question

Full size image

4.5 Correlations between students’ perspectives and academic achievement

Correlations between students’ perspectives and academic achievement are shown in Fig. 6. The formative scores had positive correlations with “enthusiasm” (R = 0.147) from Kahoot-I; “attention” (R = 0.142), “understanding” (R = 0.169), “enthusiasm” (R = 0.185), “happiness” (R = 0.222), “interested” (R = 0.160), “following” (R = 0.225), and “liking” (R = 0.173) from Kahoot-T; and “participation” (R = 0.165), “happiness” (R = 0.172), and “asking/answering” (R = 0.165) from Google-OEQ (P < 0.05 all) and had a trend of positive correlations with ‘attention’ from Google-OEQ (R = 0.127, P = 0.073) and “liking” from overall ARS (R = 0.139, P = 0.050) (Fig. 6).

The summative scores were positively correlated with “enthusiasm” (R = 0.200) from Kahoot-I; and “happiness” (R = 0.164) and “liking” (R = 0.166) from Kahoot-T (P < 0.05 all); and tended to have positive correlations with “enthusiasm” from Kahoot-T (R = 0.124, P = 0.081); “attention” (R = 0.136, P = 0.053), “enthusiasm” (R = 0.130, P = 0.064), “happiness” (R = 0.120, P = 0.088), and “asking/answering” (R = 0.133, P = 0.058) from Google-OEQ; “enthusiasm” from Google-MCQ (R = 0.133, P = 0.058); and “enthusiasm” from overall ARS (R = 0.125, P = 0.076) (Fig. 6).

Quartiles of the summative scores had positive correlations with “enthusiasm” (R = 0.230) from Kahoot-I; “happiness” (R = 0.170) and “liking” (R = 0.185) from Kahoot-T; “attention” (R = 0.148), “enthusiasm” (R = 0.157), “happiness” (R = 0.148), and “asking/answering” (R = 0.145) from Google-OEQ; “attention” (R = 0.152) and “enthusiasm” (R = 0.172) from Google-MCQ; and “enthusiasm” (R = 0.140) from overall ARS (P < 0.05 all); and had a trend of positive correlations with “enthusiasm” from Kahoot-T (R = 0.126, P = 0.076) and “liking” from overall ARS (R = 0.122, P = 0.083) (Fig. 6).

Fig. 6
figure 6

Correlations between students’ perspectives and academic achievement

Full size image

4.6 Multiple regression analysis

Multiple regression analysis for formative scores, summative scores, and the percentile of summative scores is shown in Table 3. When formative scores were set as the dependent variable, 5 models of significant interactions were identified, including with “happiness” for Kahoot-T (R = 0.264) (model 1); or “happiness” for Kahoot-T and “liking” for Google-MCQ (R = 0.398) (model 2); or “happiness” for Kahoot-T, “liking” for Google-MCQ, and corrected scores from Kahoot-I class 3 part 3 (R = 0.438) (model 3); or “happiness” for Kahoot-T, “liking” for Google-MCQ, corrected scores from Kahoot-I class 3 part 3, and “happiness” for Google-MCQ (R = 0.465) (model 4); or “happiness” for Kahoot-T, “liking” for Google-MCQ, corrected scores from Kahoot-I class 3 part 3, and “happiness” for Google-MCQ, and “understanding” for Google-MCQ (R = 0.487) (model 5) as the independent variables (P < 0.01 all) (Table 3).

For summative scores as the dependent variable, 4 models of significant interactions were identified, including with “concept” for Kahoot-T (R = 0.271) (model 1); or “concept” for Kahoot-T and “understanding” for Google-MCQ (R = 0.366) (model 2); or “concept” for Kahoot-T, “understanding” for Google-MCQ, and corrected scores from Kahoot-I class 3 part 3 (R = 0.405) (model 3); or “concept” for Kahoot-T, “understanding” for Google MCQ, corrected scores from Kahoot-I class 3 part 3, and “follow” for Kahoot-T (R = 0.438) (model 4) as the independent variables (P < 0.01 all) (Table 3).

By setting the percentile of summative scores as a dependent variable, 4 models of significant interactions were identified, including with class 3 posttest score (R = 0.276) (model 1); or class 3 posttest score and “concept” for Kahoot-T (R = 0.345) (model 2); or class 3 posttest score, “concept” for Kahoot-T, and “understanding” for Google-MCQ (R = 0.402) (model 3); or class 3 posttest score, “concept” for Kahoot-T, “understanding” for Google-MCQ, and class 2 posttest score (R = 0.441) (model 4) as the independent variable (P < 0.01 all) (Table 3).

Table 3 Multiple regression analysis for formative scores, summative scores, and the percentile of summative scores
Full size table

In addition to a Likert scale, students provided positive and negative comments (with number of comments) as followings: “Using ARS was fun (41 responses); I liked the class using ARS (18 responses); I was not sleepy (14 responses); I was not bored during class (10 responses); The activities was interesting (9 responses); The activities made me understand the teaching content (6 responses); I could participate in the class (6 responses); I liked peer learning (4 responses); I was not stressful (3 responses); and I felt enthusiastic (2 responses)”. Furthermore, students were impressed with using Google-OEQ to ask questions or give comments/feedback to the instructor during class and would like to have this system in every class.

The major shortcomings of using ARS in a classroom were poor internet signal (20 responses) and insufficient projector illumination (4 responses). Thus, in order to use ARS in class effectively, the infrastructure especially good internet signal and audio-visual equipment are ones of the major concern factors.

A summary of results regarding the top three ARS formats for each learning aspect determined by satisfactions/benefits is shown in Fig. 7.

Fig. 7
figure 7

The top three ARS formats for each learning aspect determined by satisfactions/benefits. ARS: Audience response systems, Kahoot-I: Kahoot for individual play, Kahoot-T: Kahoot for team play, Google-MCQ: Google Forms and Google Sheets with multiple-choice question, Google-OEQ: Google Form and Google Sheets with open-ended question, *the same rank

Full size image

5 Discussion

The present study determined students’ perspectives regarding the use of many ARS formats in different aspects with or without subgroup analysis into quartiles of the summative score. Furthermore, this study also determined correlations between students’ perspectives and academic outcomes. To the best of our knowledge, this is the first study that compared students’ perspectives between different ARS formats with or without subgroup analysis of students into quartiles of the summative score.

For all ARS formats, students rated the highest scores for “participation”, followed by “enthusiasm”. Students rated scores more than 4 out of 5 for almost all aspects except “understanding” for Kahoot-I. These results were in accordance with previous studies showing that students rated scores for more than 4 out of 5 for questions asking whether ARS effectively promote teaching and learning (Beekes, 2006; Caldwell, 2007; Elliott, 2003; Guarascio et al., 2017; Lee & Dapremont, 2012; Porter & Tousman, 2010). Overall, ARS enhances teaching and learning as it allows students to actively participate and response to questions; livens up a classroom; increases alertness; avoids student unwillingness from fear of public mistakes or embarrassment by being anonymity; makes them feel that they are not the only one who answered wrong (Beatty et al., 2006); and causes them to be more likely to ask and answer questions (Caldwell, 2007).

Unsurprisingly, students rated the lowest scores for “understanding” for all ARS formats as a previous study reported a little influence of ARS on the understanding of the teaching content (Dhaliwal et al., 2015). Since understanding depends on many factors, including prior knowledge, intellectual ability, and attention span, using ARS as an interactive teaching aid might less enhance students’ understanding compared to other aspects.

Remarkably, scores for “enthusiasm” for Kahoot-I was rated highest in the Q4 group compared to other groups and had positive correlations with the formative, summative, and quartiles of the summative scores. These results indicate that higher academic achieved students felt more enthusiastic when doing test individually compared with other students and scores for “enthusiasm” had a positive influence on academic achievement. This might be because higher academic achieved students like individual competition as they might be able to handle competition better than their peers (Firmin et al., 2009). A previous study revealed that enthusiasm, one of the emotional dimensions of engagement (Skinner et al., 2008), causes a positive impact on academic outcomes (Reyes et al., 2012). Thus, enhancing students’ enthusiasm should be promoted during class to augment students’ academic achievement.

For Kahoot-T, the Q1 students rated lower scores for “happiness”, “liking”, and “understanding” than their peers suggesting that low academic achieved students felt less happiness, less liking, and less understanding, for team competition than other students. This is probably because they could not follow their peers during group learning.

Kahoot-T was used in the class which required students to prepare and review for known knowledge. Low acdemic-achieving students, Q1 students, may face challenges with this ARS method and could not effectively appreciate external feedback which cause them to feel less happy, less engaged, and have a poorer understanding of team competition than other students. In comparison, students with higher academic achievement may have enough aptitude or certain learning habits that promote self-regulated learning. The higher and adequate preparation of these students enables them to be aware of their own knowledge, beliefs, and cognitive skills and could meaningfully interpret feedback from Kahoot-T (Nicol & Macfarlane-Dick, 2006).

Furthermore, for Kahoot-T, the formative scores had positive correlations with “attention”, “understanding”, “enthusiasm”, “happiness”, “interested”, and “liking”; while summative and quartiles of summative scores had positive correlations with “happiness” and “liking” and had a trend of a positive correlation with “enthusiasm”. These results implied that “happiness” and “liking” of students from using Kahoot in the team mode correspond with academic outcomes.

For Google-OEQ, scores for “participation”, “happiness”, and “asking/answering” were positively correlated with the formative scores while scores for “attention”, “enthusiasm”, “happiness”, “asking/answering” were positively correlated with quartiles of the summative score. For Google-MCQ, scores for “attention” and “enthusiasm” had positive correlations with quartiles of the summative score. For overall ARS, scores for “enthusiasm” were positively correlated with quartiles of the summative scores and tended to have a positive correlation with the summative score; and scores for “liking” tended to have positive correlations with the formative scores and quartiles of the summative score. These results together indicate that students who had higher academic performance rated higher scores for various aspects of ARS implying that high academic achieved students liked ARS more than low academic achieved students. Since ARS was used as an active learning tool, it could probably be interpreted that students who got higher scores preferred the interactive learning technique than students who got lower scores. As a result, implementation of ARS as an interactive tool in teaching and learning processes should be concerned among different backgrounds of students. Some of them might not be able to catch up with the activity during class. Thus, the appropriate teaching pace should be concerned and learning materials should be well prepared and provided to support those struggling students.

When compared between different ARS formats, scores for Kahoot-T were rated highest for “enthusiasm”, “happiness”, “liking”, and “competition”. In this study, Kahoot-T was the highest rated format which might be because friendly peer competition in groups could make students be enjoyable and enthusiastic during class (Meng et al., 2019), explore alternative viewpoints, and ask for and hear different explanations (Caldwell, 2007). Peer discussion could augment students’ understanding as explanation from other students help them learn and feel engaged during class (Caldwell, 2007).

For “understanding”, students rated higher scores for Google-MCQ and Google-OEQ than Kahoot-I and/or Kahoot-T. By using Google-MCQ and Google-OEQ, students were given more time to answer the questions (around 30–60 s per question) compared with Kahoot-I and Kahoot-T (around 15–20 s per question). A previous study revealed that a longer period of time allocation for each question increased students’ performances (Hofmeister, 2018). Thus, more time providing for students in activities could probably lead to more understanding.

Although Google-OEQ was voted about 20% in all aspects, it was voted most for “understanding” suggesting that this form of ARS could fill the gap in enhancing students understanding.

“SimilarPrePost” was rated highest in all aspects, including “attention”, “understanding”, “enthusiasm”, “happiness”, and “liking” compared to other formats. Pretest questions help students recall prior knowledge learned before class, enabling them to connect it with new information (Hew & LO, 2018). Furthermore, properly designed pretest questions related to key principles of the teaching content could potentially help students capture the concept of the class more effectively. In some group activities, since the posttest scores were included in the total scores of the subject, the similarity between pretest and posttest questions could reduce stress among students, as they might not be anxious about what would be asked in the posttest questions. However, in some classes, pretest questions were different from posttest questions. This inconsistency was designed in order to prevent students in ignoring other content rather than that appeared in the pretest questions. Collectively, using ARS could augment students’ learning; however, employing ARS with this particular objective, along with the proper teaching plan, could further enhance the benefits of using ARS in a classroom.

In the regression analysis for formative scores, factors contributing to these scores included “happiness” for Kahoot-T, “liking” for Google-MCQ, corrected scores from Kahoot-I class 3 part 3, and “happiness” and “understanding” for Google-MCQ. Since the formative questions involved immediate recall or simple critical thinking questions that required attention during class, it’s not surprising that the factors contributing to formative scores encompassed both the emotional dimension of student engagement, such as happiness and liking, and the cognitive dimension, including corrected scores and “understanding”. The scores from Kahoot-I class 3 part 3 were included in the model, possibly because the questions in this part focused on critical thinking. For summative scores, factors contributing to these scores included “concept” for Kahoot-T, “understanding” for Google MCQ, corrected scores from Kahoot-I class 3 part 3, and “follow” for Kahoot-T, all reflecting the cognitive dimension of students’ engagement. Interestingly, “concept” and “follow” for Kahoot-T were additions to the cognitive dimension of student engagement from the formative score models, highlighted the necessity for students to understand and conceptualize content for summative scores. These factors required deeper application and analysis, indicative of the higher-order thinking skills essential for summative assessments. For the percentile of summative scores, factors contributing to the regression models included class 3 posttest score, “concept” for Kahoot-T, “understanding” for Google-MCQ, and class 2 posttest score, all reflecting the cognitive dimension of student engagement. Since two posttest sessions used the same questions as the pretest, understanding at the end of the classes was a crucial factor in determining students’ higher ranking in the percentile of summative scores.

5.1 Limitations

The limitations of this study include: (1) we have recruited only ARS formats used in five classes of one subject, which may not have included all the ARS formats in use; (2) we did not conduct focus group interviews because our questionnaires yielded comprehensive data that met our objectives. To strengthen future research, it would be beneficial to explore the integration of focus group interviews to complement the findings obtained through questionnaires. Furthermore, extending the implementation of these ARS formats across multiple classes would enable a more holistic analysis, thereby enhancing the understanding of the overall benefits of ARS for the entire program.

6 Conclusion

Students agreed that using ARS in class could promote their learning in various aspects. Using Kahoot in the team mode was rated highest among many ARS formats. Low academic-achieving students rated lower scores in various aspects than high academic-achieving students. For understanding, Google-MCQ with self-pacing was the most suitable format among other formats. Using ARS with the particular objective, having similar pretest and posttest questions, could enhance students’ satisfaction than using ARS alone. Formative scores were influenced by both emotional and cognitive dimensions of student engagement, whereas summative scores and the percentile of summative scores were primarily influenced by the cognitive aspect of student engagement. Thus, it is important to appropriately select the ARS format(s) to maximize the achievement of the learning objectives.