1 Introduction

The rapid developments in artificial intelligence (AI) technologies in recent years have profoundly affected all areas of society, including the economy, politics, science, and education (Luan et al., 2020). AI technologies, which are increasingly prevalent in education, hold great promise for enhancing students’ learning performance and experiences, as well as teachers’ instructional practices (Hwang et al., 2020). Additionally, these technologies are increasingly used by educators at all levels of education (Chen et al., 2020a). It is predicted that the influence of AI in education will increase even more in the coming years (Gašević et al., 2022). Munde (2023) stated that in the education market industry, the monetary value of AI is expected to undergo an annual growth rate of 38% from 2023 to 2030, rising from US$ 3.45 billion to US$ 23.82 billion during this period. It is believed that this growth will significantly change the traditional patterns of education and create new dynamics for the education stakeholders (Giannini, 2023).

Today, various AI technologies are being employed in educational environments. Some of these tools include adaptive/personalized learning systems that customize the learning process by analyzing students’ interests and needs, automatic assessment systems that help teachers analyze students’ level in a particular subject area, and AI-supported self-assessment systems that provide feedback and assessments to students without time and place restrictions. These technologies contribute to the learning processes with their advantages such as instant feedback, flexible and personalized learning experiences. They also enable more effective teaching, bringing a series of benefits that are difficult to obtain otherwise (Akgun & Greenhow, 2022; Kabudi et al., 2021; Yang et al., 2022). AI-powered chatbots, on the other hand, have been widely employed in pedagogical fields such as language education and computer science for many years (Zhang et al., 2023). Studies on this subject have revealed the concept of Chatbot-Mediated Learning (CML) in the literature. (Winkler & Söllner, 2018). However, research on using chatbots in education is limited to only a few academic disciplines (Wong, 2022). Social Studies education stands out as one of the areas where no studies are currently carried out on this subject. In this context, it can be said that there is a need for studies focusing on the use of AI-supported chatbots in Social Studies education from a broad perspective. At this point, it is thought that the relevant study will be useful in filling the relevant gap. In the continuation of the study, the literature on the use of chatbots in education was first evaluated, and the problem case of the research was detailed. Then, the method of the study was explained. Following that, the findings of the research were presented. Finally, the relevant findings were discussed, and suggestions for practice and future research were made.

2 Literature review and problem case

2.1 Chatbots and their educational use

Today, chatbots stand out as a novel AI technology widely employed in the field of education. According to a simple definition, chatbots are computer programs designed to simulate human speech through text or voice interactions (Brush & Scardina, 2021). Alternatively, they can be described as artificially constructed software using natural language as input and output to communicate with humans (Wang et al., 2021). The primary purpose of a chatbot, which can participate in written or spoken conversations, is to simulate intelligent human speech. Thus, the person can have the opportunity to experience a real-like chat process. In the literature, various names are given to chatbots, such as AI assistants, smart virtual assistants, digital assistants, speech agents, and virtual agents (Molnár & Szűts, 2018; Surendran et al., 2020).

Chatbots have been traditionally used in areas such as customer services, finance, marketing, technical support, e-trade industry for many years (Hasal et al., 2021). However, their use in learning-teaching processes has recently gained popularity (Yin et al., 2020). Studies on this subject have revealed the concept of chatbot-mediated learning (CML) in the literature. This type of learning, which can be seen as an extension of technology-based learning, points to the positive effect of chatbots in learning-teaching processes (Winkler & Söllner, 2018). Okonkwo and Ade- Ibijola (2021), who evaluated the existing literature on the use of chatbots in education, found that chatbots are primarily used in education with their function of improving teaching and learning. Pérez et al. (2020), on the other hand, classified chatbots as “Service-oriented chatbots” aimed at facilitating staff-student interaction and “Teaching-oriented chatbots” to convey information to students about a particular subject. It was emphasized in the study that there has been an increasing interest in the use of teaching-oriented chatbots. Similarly, Zhang et al. (2023) revealed that the most common pedagogical purpose of chatbots is to present a learning material. Hwang and Chang (2023) conducted a comprehensive study examining the trends in studies on the use of chatbots in education. As a result of the study, it was determined that the relevant studies mainly focused on language education, and the studies within the scope of K-12 education were insufficient. In addition, it was found that most of the studies did not employ a specific learning strategy, and they were planned according to the ‘Guided learning’ approach, in which students practiced with a chatbot under a teacher’s guidance.

Various studies in education have confirmed the benefits of chatbots. A quantitative study on the use of chatbots in language education was conducted by Kim (2018). As a result of the relevant study, it was determined that the learning performance and attitudes of students toward learning English vocabulary were positively affected by chatbots. In a different study, it was found that the English grammar skills of students in the experimental group, who learned by interacting with a chatbot, were significantly higher than those of the control group students, who learned by interacting with a human partner (Kim, 2019). In the study conducted by Hsu et al. (2021), it was revealed that chatbots positively and significantly affected students’ English speaking skills. On the other hand, Vázquez Cano et al. (2021) tested the effect of chatbots on punctuation learning. It was found in the study that the academic achievement of the experimental group students, who learned punctuation marks with a chatbot, was significantly higher than that of the control group students. Chuah and Kabilan (2021) revealed that English teachers regarded chatbots as easy-to-use applications that help teach vocabulary and grammar. Jeon (2022) conducted a study with thirty-six Korean primary school students learning English as a foreign language. As a result of the study, it was determined that participant students saw chatbots as easy and enjoyable tools to use, but they stated that the chatbot could not interact as well as a teacher. In some cases, the chatbot could not make sense of the inputs from the students, and the students found this worrying.

In a different study, Pereira (2016) developed a chatbot named Dawebot for 23 university students in the computer science department. After the lessons, the students answered multiple-choice questions asked by the chatbot, testing their understanding of the material covered in class. As a result of the study, 89% of the students stated that using chatbots in Q&A activities was a good idea. Students generally found chatbots easy to use, indicating that their interest increased in the lesson. On the other hand, in the study conducted by Bii et al. (2018) it was determined that teachers who engaged in chatbot-assisted teaching activities found it useful and enjoyable to use chatbots in computer classes. Yin et al. (2020) conducted a study to examine the effects of a chatbot-mediated teaching process on students’ learning motivation and performance in computer science. The quasi-experimental design was used in the study. At the end of the study process, it was found that the learning motivation of the students in the experimental group who learned through chatbots was significantly higher than that of the students in the control group, who learned through the traditional method.

An exemplary study on the use of chatbots in mathematics education was conducted by Grossmann et al. (2019). It was stated in the study that online mathematics education generally lacks basic features of face-to-face teaching, such as personalized feedback. To simulate this type of interaction, the researchers developed a chatbot that explained mathematical concepts, offered students practice questions, and provided specific feedback. In the studies by Chen et al. (2020b) and Reiss (2021) the contribution of AI technologies to individuals in need of special education was emphasized. Similarly, attention was drawn to the potential of chatbots for special education processes in the study conducted by Mateos Sanchez et al. (2022). Within the scope of the research, a chatbot-mediated mobile application called CapacitaBOT was developed to provide intellectually disabled individuals with various social skills. On the other hand, Padilha et al.‘s (2022) study focused on the contributions that chatbots can offer to visually impaired students. In this context, a chatbot called ELIOT was developed to teach software engineering to visually impaired students. As a result of the pilot application, it was determined that the chatbot had the potential to respond to students quickly and effectively.

When the literature is analyzed in general, it is evident that chatbots make significant contributions to academic success by positively transforming students’ learning processes. Additionally, chatbots make lessons more exciting and enjoyable for students, serving as an easy-to-use and efficient tool for instructors during the teaching process. However, as pointed out in various studies, it can be observed that studies at the K-12 level are insufficient. Moreover, related studies generally focus on the quantitative or qualitative aspects of the process, primarily emphasizing student experiences. Additionally, it was observed that existing studies examine the impact of chatbots on post-test academic achievement rather than their influence on permanent learning. Furthermore, it is evident that relevant studies are widespread in limited fields such as language education and computer science. However, there is currently no study in the field of Social Studies education. At this point, it is thought that this study will be helpful in addressing the identified deficiencies determined based on the current literature.

2.2 Problem case

Today, technology is widely used in Social Studies teaching-learning processes, along with alternative education methods. Technological tools make learning Social Studies easier and more efficient for students. They are employed as supportive instruments by teachers who enjoy incorporating them into their classes. When the historical use of technology in Social Studies is analyzed, it can be seen that they have been used as tools contributing to Social Studies education for many years (Hicks et al., 2014). However, some educational technologies used in different disciplines haven’t been applied in Social Studies education. One of these technologies is chatbots. When the literature is examined, it can be seen that chatbots have been widely used in fields such as language education and computer science, especially in the last few years (Hwang & Chang, 2023). However, no study has been carried out specifically for Social Studies education. Considering the rise of AI-supported chatbots as a tool that improves learning-teaching processes and the positive contributions of this technology to aspects such as academic success, permanent learning, and motivation (Hsu et al., 2021; Kim, 2018; Lee et al., 2011; Tham & Ruan, 2019; Yin et al., 2020), it can be argued that it is necessary to eliminate this gap and reveal how this novel technology will transform Social Studies education.

In addition, technological instruments play a crucial role in Social Studies education (Lee & Friedman, 2009). Chatbots, on the other hand, have high potential for creating interactive learning environments, providing personalized and autonomous learning experiences, offering instant accessibility and feedback opportunities, enabling access to education outside of school hours, and facilitating rich learning-teaching experiences that engage multiple senses (Chocarro et al., 2021; Clarizia et al., 2018; Fryer & Carpenter, 2006; Wong, 2022). All of these indicate that this technology might contribute to Social Studies education from different aspects. In this sense, it can be argued that there is a need for studies that highlight the potential benefits of chatbots in Social Studies Education. Another important requirement in the field is preparing studies that guide Social Studies educators about the use of AI-supported chatbots. As known, the competence of Social Studies educators in terms of integrating technology into education is crucial (Fontana, 1997). For a transformative and effective Social Studies education, Social Studies educators should adopt a comprehensive view of technology integration. This is possible only when they understand how to use novel and newly developing technological tools effectively (Doolittle & Hicks, 2003). In this respect, it can be said that it is important to conduct studies that will provide a basic understanding to Social Studies educators about using chatbots in education.

As a result, the absence of studies on the use of chatbots in Social Studies education, along with the lack of a theoretical framework guiding Social Studies educators in integrating chatbots into their instructional practices, can be regarded as limitations that hinder the adoption of this technology in Social Studies education. In this regard, it can be argued that there is a need for studies comprehensively addressing the process of using chatbots in Social Studies Education. Therefore, it is believed that the present study will be useful in addressing the relevant needs.

The aim of this research is to investigate the educational potential of AI-powered chatbots in Social Studies learning-teaching processes. In this context, the research aims to answer the following questions.

  1. 1.

    Do the chatbots significantly affect the students’ academic success in Social Studies classes?

  2. 2.

    Do the chatbots significantly affect the students’ permanent learning in Social Studies classes?

  3. 3.

    How did students experience chatbots in the process of learning Social Studies?

  4. 4.

    How did the practitioner teacher experience chatbots in the process of teaching Social Studies?

3 Method

In this section, research design, study group, data collection techniques/tool, data analysis, development of chatbots, and process/application subheadings are presented.

3.1 Research design

This research was conducted according to the embedded design, evaluated within the framework of mixed methods research. In this research design, the study is primarily either qualitative or quantitative. However, data obtained through alternative methods are necessary to support, generalize, and explain the findings (Plano Clark & Creswell, 2008). The primary purpose of this study is to examine the potential of chatbots in Social Studies classes using a quantitative research methodology. Furthermore, a qualitative process is conducted to present the teacher and student experiences with the process and chatbot technology. By employing this comprehensive approach, the study aims to provide a more thorough and insightful explanation of the quantitative process. In this regard, the quantitative dimension of the research was carried out according to the pretest-posttest control group design, which is one of the quasi-experimental research designs. In this design, groups cannot be created using the random method; however, existing groups can be matched based on predetermined variables (Büyüköztürk et al., 2017). In this study, existing groups are matched according to the pretest scores and assigned as experimental and control groups. The qualitative dimension of the research was carried out according to the “basic qualitative research” design. This qualitative research design, often used in the field of education, aims at determining how participants interpret their experiences (Merriam, 2009). In this context, semi-structured interviews were conducted with students and practitioner teachers regarding the process. Thus, an attempt was made to elaborate on the quantitative aspect of the study using the obtained qualitative data.

3.2 Study group

The study group for this research comprises a total of 78 6th-grade students from three different classes and one teacher who conducted the practices and supervised the process. All students who took part in the research completed the entire measurement process, including the pretest, posttest, and permanence test. Additionally, 17 volunteer students from the experimental group participated in the qualitative aspect of the research, providing their opinions on the process and chatbot technology. The distribution of study group students who completed all the processes is presented in Table 1.

Table 1 Characteristics of the study group
Full size table

Furthermore, another participant in the study group of the research is Fırat, the teacher who conducted the experimental process. His willingness to participate is one of the reasons he was chosen. Additionally, his statement about regularly incorporating technology-based teaching tools in the classroom and expressing familiarity with technology use was pivotal in his selection.

3.3 Data collection techniques/tools

The study was designed around four fundamental questions. The primary data collection tool for the first and second questions in the research is the “6th Grade Culture and Heritage Success Test,” developed by the researchers, consisting of 33 items. The internal consistency, calculated with KR20, revealed a very high result of 0.86. The general difficulty level was 0.49, indicating a medium level, while the general index of distinctiveness was 0.46, signifying a very good level. On the other hand, qualitative data for the study were collected with semi-structured interview forms developed by the researchers.

3.4 Data analysis

Different qualitative and quantitative data analysis methods were employed in analyzing the obtained data. The data analysis methods used are presented in Table 2.

Table 2 Basic data collection techniques/tools and data analysis methods
Full size table

As shown in Table 2, quantitative data analysis was employed for the first and second questions of the research. Descriptive statistics were utilized to present the general states of groups regarding the tests. One-Way ANOVA was used to determine the equalization between the pretest scores of groups. Although there wasn’t a statistically significant difference between the pretest scores of groups, small differences in these scores might affect the subsequent results (Büyüköztürk, 2007). Therefore, One-way covariance analysis (ANCOVA) was used to compare the posttest and permanence test scores. Normalcy tests and Levene homogeneity test were used to test the assumptions of One-Way ANOVA and One-Way ANCOVA. Bonferroni multiple comparison tests were utilized to determine the groups that had differences, as revealed by ANCOVA. The partial eta-square value (η2) was calculated to reveal the effect size of the results obtained with the posttests and the permanency test. The Statistical Package for the Social Sciences (SPSS) 28 was employed in all these processes.

On the other hand, the qualitative data of the research were analyzed using the inductive content analysis method. In this method, the analysis process starts at a small point and evolves into a comprehensive understanding. The hidden patterns, categories, and themes within the datasets are discovered and presented as a meaningful whole (Patton, 2002). In this context, the data obtained through interviews were analyzed with the inductive content analysis method, and codes were prepared; subsequently, categories and themes were created. Throughout this process, peer debriefing, member checking, diversification, and prolonged involvement methods were employed to ensure the credibility of the research (Yıldırım & Şimşek, 2016). Additionally, in the qualitative findings, participants’ opinions were presented using pseudonyms.

3.5 Development of chatbots

One of the most important stages of the study is the development of chatbots. This process was carefully designed, and a specific chatbot creation and development plan was followed. The chatbot development plan for the process is presented in Table 3.

Table 3 The basic framework of developing chatbots
Full size table

As can be seen in Table 3, a single chatbot named “Sosyalcibot” was developed, and the chatbot’s existence was represented in robot form. Additionally, Sosyalcibot’s social presence was tried to be developed with a fictional story about itself. Sosyalcibot was developed within the scope of the “Culture and Heritage” learning domain, comprising 5 acquisitions in 6th-grade Social Studies. SosyalciBot was developed according to the closed domain and retrieval model, which is frequently used in educational research (Kim et al., 2022; Liu et al., 2020). With the retrieval model, efforts were made to train the chatbot more easily and reduce the possibility of grammatical errors and meaningless answers (Kojouharov, 2016; Nguyen et al., 2021; Kim et al., 2022). By choosing the closed domain, attempts were made to offer a narrower but more efficient and structured chat experience (Liu et al., 2020; Kim et al., 2022). At this point, the function of the chatbot was limited to teaching the basic content of the 6th-grade Culture and Heritage learning area and conducting Q&A activities supported by instant feedback and clues.

A hybrid chatbot model, combining AI-supported smart chatbots with button-based simple chatbots, was utilized in the study. These chatbots engage with users in a predefined manner, yet they employ AI to discern user intentions and interpret messages (Trofymenko et al., 2021). The chatbot’s AI system was grounded in natural language understanding, considered a subset of natural language processing (Jung, 2019; Kidd & Saxena, 2021). Dialogflow, Google’s natural language understanding platform, was employed for this purpose. Additionally, button-based feedback features were incorporated into the chatbot to address off-topic messages and significant spelling errors. Mindbehind chatbot development platform was employed throughout all stages. It integrated various language processing solutions and was built using technologies such as Kubernetes, MongoDB, and React, operating on the Google Cloud. The pedagogical content and connections that underpin SosyalciBot were constructed using this platform. The chatbot was connected to Dialogflow through the relevant platform and trained. SosyalciBot was presented to users on https://www.chatbotlaogren.com/. The homepage of the site and platform’s chatroom page is presented in Fig. 1.

Fig. 1
figure 1

Homepage of the Website/ platform’s chatroom page

Full size image

During the development process, one written and one audio version were created for each acquisition. The audio version, especially, was thought to be efficient in the classroom. Besides that, SosyalciBot’s lecturing and Question and answer (Q&A) modules was enriched with images, photographs, and emojis following ethical principles, and the chatbot was represented with an avatar that symbolized itself in a colorful and user-friendly interface. Some specific points were taken into consideration while designing the chatbot. The first point was integrating the chatbot’s design into constructivist Social Studies environments. At this point, rather than completely guiding the learning-teaching process, it was aimed to develop the chatbot as digital technology that supports the teacher from time to time. For this, the chatbot’s lecturing module was limited to teaching key points regarding acquisition with visuals over specific topics based on the student’s preferences. Additionally, the Q&A module was designed with a focus on activating students. For this, instead of showing the correct and incorrect answers at the end, students were given an opportunity until they figured out the correct answer. Thus, the module was designed to provide instant feedback, clues, additional information about incorrect answers, and motivating speeches. The module also included various types of questions, such as open-ended, fill-in-the-blank, and multiple-choice questions. On the other hand, a human-centered AI understanding, based on approaching AI designs through the human eye, was employed in the process (Yang et al., 2021). For this, a kind and motivating language that keeps competition in the background was used. In addition to all these, ethical elements such as algorithmic bias and transparency (Remian, 2019) were taken into consideration.

In addition to all of these, as stated in Table 3, certain steps were followed in the development of the chatbot. For this purpose, in the initial stage, themes were determined for each acquisition. Then, algorithms/content for each theme were prepared. These themes became the topics that SosyalciBot presented to students in lectures and Q&A modules after programming. The developed chatbot was evaluated with a pilot scheme in the first month of the 2021–2022 fall semester. At this point, SosyalciBot was introduced to five 6th-grade students and two Social Studies teachers. At the end of the one-week process, students evaluated the chatbot in terms of language and understandability, and teachers evaluated it regarding its suitability for the program content. The received suggestions were assessed, and the necessary adjustments were made to the chatbot.

3.6 Process/application

The first stage of the process was the implementation of pre-tests. Following the implementation of pre-tests on the students, the test data were analyzed, and the classes were randomly assigned as experimental and control groups. One week later, the practitioner teacher conducted a 3-hour pilot study on the first acquisition in a different class that was not included in the pre-tests. The purpose of this pilot study was to observe several situations: observing whether the process aligned with the lesson plans, determining the duration required for the implementation of SosyalciBot, and assessing potential unforeseen issues related to factors such as the sound quality of the smart board, internet speed, continuity of the chat flow, and the proper functioning of the system. After the end of the process, it was determined that the use of SosyalciBot took more time than expected; this situation was taken into consideration in the plans for upcoming lessons. Following this, the actual implementation process began. The teacher and the researcher collaborated to prepare lesson plans throughout the process. In-class applications were entirely conducted by the teacher. This approach aimed to mitigate researcher bias, a significant validity threat in experimental research (Fraenkel & Wallen, 2006). Moreover, the researcher actively participated in the experimental process and conducted structured observations. In the experimental group, the teacher’s implementation level of the steps outlined in the lesson plan was observed. In the control groups, it was observed whether the teacher utilized SosyalciBot along with other technologies and whether he taught the content in coordination with the experimental group. During this period, classes in the Experimental and Control 2 groups continued face-to-face. In Control Group 1, classes were conducted online for two weeks due to the pandemic, and the researcher remotely observed these classes. In the experimental process, SosyalciBot was used for two different functions. In the 1st, 2nd, and 3rd acquisitions, the chatbot was utilized to reinforce the acquired information. In these acquisitions, the chatbot re-taught the subject after the teacher and then engaged in Q&A activities with the students. On the other hand, in the 4th and 5th acquisitions, the chatbot was used at the beginning of the acquisition to prepare students for the subject, and at the end of the acquisition for a competition activity. At this point, in the 4th and 5th acquisitions, the chatbot took the role of teaching the students the subject before the teacher. following the chatbot’s instruction, the teacher provided an additional explanation of the subject. Once the teacher’s lecture concluded, Q&A module was implemented among the students in a competitive format. Images of the process are presented below (Fig. 2).

Fig. 2
figure 2

Images of the process

Full size image

After all acquisitions were taught, post-tests were implemented for all groups, and semi-structured interviews were held with 17 volunteer students in the experimental group. Additionally, five different interviews were conducted with the teacher throughout the process. Five weeks after the post-test, the same test was implemented to the groups as a permanency test, and the process was completed.

4 Findings

In this section, quantitative findings obtained from the experimental process of the study were presented. Following this, qualitative findings arising from the semi-structured interviews, conducted both throughout and at the end of the study, were presented.

4.1 Findings about the pretest scores of experimental and control groups

The first phase of the experimental process involved comparing the pretest scores of groups and testing their equivalence. At this point, One-Way ANOVA assumptions were tested (Can, 2019). Following the normalcy analyses, it was determined that normalcy assumptions were met in the experimental group (Shapiro-Wilk: p: .453; Skewness: .456; Kurtosis: .025), control group 1 (Shapiro-Wilk: p: .185; Skewness: -.106; Kurtosis: -.962), and control group 2 (Shapiro-Wilk: .526; Skewness: .352; Kurtosis: -.280). On the other hand, it was determined that the equality of variances, another assumption of ANOVA, was satisfied (p: .530). (Tabachnick & Fidell, 2013; Seçer, 2015). After confirming that the relevant assumptions were met, One-Way ANOVA was conducted to compare the pretest scores. The analysis results are presented in Table 4.

Table 4 One-way ANOVA results of pretest scores of groups
Full size table

When Table 4 is examined, it can be seen that there is no statistically significant difference between the academic success of students in the experimental and control groups before the application (F(2−75) = .325; p = .723). This result indicates that the academic success levels of students before the experimental process were similar, and equivalence was fulfilled.

4.2 Findings about the post-test scores of experimental and control groups

The post-test scores of the experimental and control groups were compared with One-Way ANCOVA. For this purpose, firstly, assumptions of One-Way ANCOVA (Can, 2019) were tested. The analyses conducted revealed that the data in the experimental group (Shapiro-Wilk: .074; Skewness: -.650; Kurtosis: -.470), control group 1 (Shapiro-Wilk: .150; Skewness: -.448; Kurtosis: -.724), and control group 2 (Shapiro-Wilk: .061; Skewness: .501; Kurtosis: -.609) followed a normal distribution. Furthermore, the equality of variances was confirmed through the Levene test (p = .063). Additionally, the linear relationship between the dependent variable and the covariate was verified through a scatter plot. Moreover, it was found that the slopes of the regression lines were equal. Thus, it was determined that all the assumptions required for ANCOVA were met. Subsequently, a one-way ANCOVA was conducted to determine the difference between the post-test scores of the experimental and control groups. Descriptive statistics for the group means are presented in Table 5.

Table 5 Corrected post-test means of experimental and control groups
Full size table

As seen in Table 5, the corrected post-test mean for the experimental group was 20,897, making it the group with the highest mean. The corrected post-test mean for Control Group 1 was 17,027, indicating that this group had the second-highest mean. The mean for Control Group 2 was 16,478, and it had the lowest corrected post-test mean. ANCOVA results regarding the significance of these differences are presented in Table 6.

Table 6 One-way ANCOVA results regarding the post-test scores of the experimental and control groups
Full size table

According to Table 6, a significant difference exists among the corrected post-test mean scores of the groups (F(2, 74) = 7.646, p = .001). In other words, there is a significant difference between the mean scores of at least two groups. The Bonferroni multi-comparison test was conducted to determine which groups exhibited significant differences. According to the results, the corrected post-test success of the experimental group was significantly higher than that of Control Group 1 (p = .007). Similarly, the corrected post-test success of the experimental group was significantly higher than that of Control Group 2 (p = .002). It was determined that there was no significant difference between the corrected post-test success scores of Control Group 1 and Control Group 2. On the other hand, the effect size of the significant result obtained with One-Way ANCOVA was analyzed, and partial eta-square (η²) was used. Accordingly, the eta-square value was .171; this value indicated that the result was both significant and had a large impact size (Green & Salkind, 2004).

4.3 Findings about the permanency test scores of experimental and control groups

One-Way ANCOVA was used to compare the permanency test results of the experimental and control groups. As the first step of the process, ANCOVA assumptions were tested. At the end of the analyses, it was observed that the data of the experimental group (Shapiro-Wilk: .368; Skewness: -.223; Kurtosis: -.827), control group 1 (Shapiro-Wilk: .829; Skewness: .117; Kurtosis: -.443), and control group 2 (Shapiro-Wilk: .077; Skewness: -.205; Kurtosis: -1,265) were normally distributed. Through the Levene test, it was found that variance equality was ensured (p = .194). The linear relationship between the dependent variable and the covariate was verified with a scatter plot. Furthermore, it was found that the slopes of the regression lines were equal. After testing all of the assumptions, One-Way ANCOVA was conducted to determine the difference between the experimental and control groups’ permanency test scores. Descriptive statistics of groups are presented in Table 7.

Table 7 Corrected permanency test means of the experimental and control groups
Full size table

As can be seen in Table 7, the group with the highest permanency score was the experimental group with a corrected mean of 21,944. Control Group 1 had the second-highest mean at 18,390, while Control Group 2 had the lowest permanency mean at 17,233. ANCOVA results regarding the significance of these differences are presented in Table 8.

Table 8 One-way ANCOVA results regarding the permanency-test scores of the experimental and control groups
Full size table

When the values presented in Table 8 are examined, it can be seen that there is a significant difference among the corrected permanency test scores of groups (F (2-74) = 9,019, p = .000). This finding indicates that there is a significant difference between the mean values of at least two groups. The Bonferroni multi-comparison test was conducted to determine which groups have significant differences. According to the results, the corrected permanency test score of the experimental group was significantly higher than that of Control Group 1 (p = .008). Similarly, the corrected permanency test score of the experimental group was significantly higher than that of Control Group 2 (p = .000). It was determined that there was no significant difference between the corrected permanency test scores of Control Group 1 and Control Group 2. Following this, the impact size of the significant result obtained with One-Way ANCOVA was analyzed. Accordingly, the partial eta-square value was .196. This indicated that the obtained result was significant, and the effect was “Large”.

As a result of the quantitative dimension of the study, it was determined that students in the experimental group, who received chatbot-assisted education, showed significantly higher success in both post-tests and permanency tests compared to the control groups. On the other hand, semi-structured interviews were conducted with the experimental group students and the practitioner teacher involved in the process. The aim was to better understand the process in which this success occurred and provide detailed insights into the role of the chatbot. The findings obtained from these interviews were summarized under two main headings: “Findings about students’ experiences” and “Findings about the teacher’s experiences”.

4.4 Findings about students’ experiences

At the end of the process, interviews were conducted with the students regarding their experiences with the process and chatbot technology. The data obtained from these interviews are presented under the themes specified in Fig. 3.

Fig. 3
figure 3

Main themes on student experiences

Full size image

4.4.1 Effects on the learning process

The theme “Effects on the learning process” aims to portray the impact of SosyalciBot on the learning-teaching processes from the perspective of the students. The categories and codes under this theme are presented in Fig. 4.

Fig. 4
figure 4

Student views in the scope of the theme “Effects on the learning process”

Full size image

Contributions to the learning performance

An important aspect highlighted by the interviewed students was the contribution of the chatbot to learning performance. Many students appreciated the chatbot’s practice of reteaching the subject after the teacher and engaging in Q&A activities. Ece expressed, “It was positive, I mean… A teacher comes and teaches a topic, and then another one comes and summarizes it”. Beyza, another participant, mentioned that there were points she couldn’t understand when taught by the teacher; however, when the chatbot reviewed the lesson, she could comprehend what she had missed. In addition, the chatbot’s practice of preparing students by teaching the subject before the teacher was also welcomed by the students. Many mentioned that it helped increase their readiness. Ece said, “It was beneficial for students who hadn’t prepared for the lesson. It served as preparation, and thanks to the chatbot, even though we didn’t fully grasp the topic, we had already learned half of it when the teacher started teaching”. Similarly, Poyraz noted that using the chatbot at the beginning of the class served as preparation and had a positive impact on their learning. On the other hand, many students stated that this practice increased their active participation in the class.

Another aspect on which students expressed positive feedback was the usability of SosyalciBot outside the school environment. Ece, Beril, and Doruk said that using the chatbot out of school enabled them to reinforce what they learned at school. Yüksel noted that he used to face difficulties studying at home, but with SosyalciBot, he had the opportunity to learn topics outside of school in a self-directed manner. Rüzgâr stated that the chatbot supported him in terms of following the lesson process and helped him feel more connected to the class. The effect of SosyalciBot on the permanency of learning was mentioned by Ece, Enes, Doruk, and Zeynep. Students stated that the chatbot contributes to permanent learning by supporting teaching with audio and visuals, providing hints in Q&A activities, and being used outside of school. The impact of the chatbot on overall success in Social Studies was also noted by several students. For example, Poyraz, Beyza, and Yüksel stated that, thanks to SosyalciBot, they achieved higher grades in Social Studies exams.

View of the lesson

Several students stated that since SosyalciBot made the process more attractive and increased their participation, they developed a more positive attitude toward Social Studies. Another frequently mentioned point by students was that SosyalciBot made Social Studies classes more enjoyable. Different students, including Enes, Doruk, Rüzgâr, Poyraz, and Onur, shared this viewpoint. Ece mentioned that history topics were boring, but with SosyalciBot, it became more enjoyable to learn them. Emre said, “SosyalciBot was enjoyable. I can tell you that, for example, a person understands a topic better if he enjoys learning it… SosyalciBot was fun, so I understood it better”. On the other hand, many students stated that this technology made classes more interesting, and this had positive effects on learning.

4.4.2 General evaluation of chatbot technology

The theme “General Evaluation of Chatbot Technology” encompasses students’ assessments of the chatbot technology. The categories and codes under this theme are presented in Fig. 5.

Fig. 5
figure 5

Student views in the scope of the theme “General evaluation of chatbot technology”

Full size image

Positive aspects

One important issue mentioned by participants was the positive aspects of SosyalciBot. Many students welcomed the fact that the chatbot has two different modules: lecturing and Q&A. On the other hand, several students said that the voice feature of the chatbot was highly beneficial. Barış mentioned that this feature could be useful, especially for visually impaired students. Öykü and Beyza said that they liked the chatbot as it was not only voiced but also sent images. Rüzgâr said that the images sent by the chatbot helped him understand where the events took place. Many students stated that in the Q&A modules of other technologies, the correct and incorrect answers are shown at the end. In contrast, SosyalciBot provides instant feedback and assists them until they reach the correct answer with hints, which is very advantageous. Another feature of the chatbot favored by students was the different question forms it used. Efecan and Özlem mentioned that as SosyalciBot could be used outside of school, especially at home, it gave them a chance to learn 24/7. While Zeynep stated that using technology at home increased her interest and motivation. Öykü, Barış, and Onur mentioned that using SosyalciBot outside of school might provide an advantage to students in case there is a disease or epidemic. Gökhan and Onur said that they liked the constructive language used by the chatbot. Gökhan stated that he liked the fact that the chatbot used some motivating expressions such as “You made it!“. About the issue, Onur said: “I mean, it never breaks hearts like a teacher, it reasks us the questions that we cannot answer correctly”. Moreover, many students welcomed the competition event held with the chatbot and mentioned that it was enjoyable and socializing. The contribution of SosyalciBot to the memorization processes was also another issue welcomed by several students such as Ece and Özlem.

Similarities with a real teacher

Another point about SosyalciBot mentioned by participants was its similarities to a real teacher. According to students, these similarities included “lecturing”, “question-answer”, “visually-supported teaching”, “feedback”, “clues”, “motivation”, and “addressing students by name”. Ece, who asserted these similarities, said, “When I make a mistake, it tells me not to worry; it gives me clues and helps me find the correct answer. It also summarizes the topics… Presents images, which is something that a real teacher does”. Apart from these, the ability of the chatbot to address the student by name in the written version was evaluated by the students as a similarity to a real teacher.

Advantages when compared to a real teacher

Students asserted the advantages of SosyalciBot when compared to a real teacher. One of the most frequently mentioned advantages was the feature of teaching a topic outside of school. On the other hand, 24/7 accessibility was another advantage mentioned by students. Ümit said, “We cannot reach a teacher 24/7, but we can reach the chatbot”. According to students, one of the advantages of the chatbot is that it is more tolerant. This feature was emphasized by Enes, Beril, Rüzgâr, and Onur. For instance, Enes said, “The advantages of the chatbot compared to a teacher are: It is less angry than a teacher, it is nicer, and helps us more when we do not know a question”. Rüzgâr and Yüksel stated that the chatbot does not get sick, which was an advantage compared to real teachers. Besides, some students mentioned that the chatbot has different instructional advantages. For instance, Emre said that real teachers might sometimes forget, but chatbots do not. Barış stated that the chatbot reacts more quickly than teachers. Yüksel said, “When our teacher teaches a subject, she/he has to be quick, but the chatbot doesn’t have such problems, it can tell us the topic as many times as we want”.

Language of the technology

Students made evaluations about the language used by SosyalciBot. The prominent points in these evaluations were that the language of technology was motivating, gentle, and anthropomorphic. According to some students, the language used by the chatbot was quite motivational. For instance, Gökhan and Emre mentioned that this motivational language of the chatbot gave them comfort and they felt psychologically good. Rüzgâr stated that thanks to the motivational language, he felt encouraged while learning. He said, “I usually used to feel ashamed and would hesitate in Q&A activities in both Social Studies and other classes, but I could answer without hesitation while using the program”. About the issue, Barış said: “I felt good as a student. I believe that my friends also felt the same way…. It says: ‘You are amazing!’ ‘Wonderful!’ or ‘Well, I’m making this question easy for you!’”. On the other hand, students underlined the gentle language used by SosyalciBot. Many students said that this polite language used by the chatbot increased their willingness to use this technology. Ümit said, “It was very nice and polite. It said, ‘All right, I will make this question easier for you…’”. Enes said, “It was polite, calm, and didn’t get angry at all…”. Besides these, some students mentioned that the chatbot’s language had some anthropomorphic characteristics, and this had some positive effects. Ece said: “Frankly, I felt like I was talking to a real teacher. It motivated me, left a good impression…”. About the same issue, Beril said: “The way it talked was nice. We felt good; it was like a real teacher…”.

4.4.3 New ideas about developing chatbot technology

The theme “4.4.3. New Ideas About Developing Chatbot Technology” includes students’ evaluations of the chatbot’s limitations and suggestions for improvement. The categories and codes under this theme are presented in Fig. 6.

Fig. 6
figure 6

Student views in the scope of the theme “New ideas for improving chatbot technology”

Full size image

Limitations and improvement suggestions

One of the most important limitations emphasized by students was that the chatbot was not sensitive to voice. Most students tended to appreciate the inclusion of a voicemail feature for the chatbot. However, they also pointed out that they had to provide answers to the chatbot only in written form. This was considered a significant drawback according to students. Beyza and Zeynep suggested that adding a voice recognition feature to the chatbot would be beneficial. According to some participants, since SosyalciBot did not physically exist in the learning environment, they found it challenging to concentrate. Additionally, they found it strange that they could only hear its sound and couldn’t physically interact with it. Doruk suggested a solution to this deficiency. He said, “I feel that it cannot control us very much. In my opinion, the mutual interaction process can be strengthened with a camera. It can follow us through a camera and see if we focus on the lesson”. Some students considered the fact that the chatbot can be accessed via the website and requires the internet as a significant shortcoming. They suggested that the chatbot should be turned into an application that can be used without the internet. Some students assessed this in the scope of digital division and said that turning the chatbot into an application could remove the limitations of reaching it.

Another limitation highlighted by students was the disadvantages arising from the AI itself. The fact that AI cannot respond to every request of the student and the limited language it uses attracted the attention of the students. For instance, Barış stated that the language of AI was different from the way that teachers spoke in classes. Enes said, “It can teach, I mean it can give us information about a subject, but it doesn’t teach as well as real teachers”. Öykü said that sometimes teachers can provide some out-of-context information and reinforce subjects by using different perspectives and resources, but the chatbot has limitations. The single concrete interpretation of how to solve this limitation was presented by Barış. He said that AI hasn’t been fully developed yet, and it will be good to improve the AI to strengthen the communication capacity of chatbots. Another limitation of the chatbot frequently emphasized by students was the lack of dynamism in the chatbot’s content. Ece and Özlem suggested that animations and videos should be incorporated to enhance its dynamism. Some participants mentioned that, while not critical deficiencies, additional features could be added to SosyalciBot to improve it. These include different language options, interesting contents such as mini-games, music, and interesting news, easy-to-difficult questions, and expanding the scope of the chatbot.

4.5 Findings about teacher’s experiences

During and at the end of the process, interviews were held with Teacher Fırat about his experiences. The data obtained are presented under the themes specified in Fig. 7.

Fig. 7
figure 7

Main themes about teacher’s experiences

Full size image

4.5.1 Teacher-chatbot cooperation

The theme “4.5.1. Teacher-Chatbot Cooperation” covers the teacher’s experiences regarding the teaching process carried out with the chatbot. The theme, along with its categories and codes, is presented in Fig. 8.

Fig. 8
figure 8

Teacher’s views in the scope of the theme “Teacher-chatbot cooperation

Full size image

Learning-teaching process

The teacher, stated that technology brought about some changes in the learning-teaching processes. He emphasized that the chatbot assisted him in improving readiness and reinforcement. During the last interview, he said “Of course, there was cooperation between me and the chatbot. For example, the chatbot taught the subject at the beginning, which increased the readiness level of students. It also covered the subject again after I had taught. This reinforced what they had learned”. According to him, the use of the chatbot at the beginning also increased their attendance and made them more active. He underlined that this attendance played a key role in their success.

He stated that SosyalciBot activated different sense organs, improved students’ active listening skills, and increased their motivation through the activity of competition. According to him, these competitions increased cooperation among students as they competed in groups. He stated that SosyalciBot made classes more attractive. He especially mentioned that the Q&A module made a substantial contribution to learning. The teacher stated that the history topics in Social Studies usually require memorization, and the chatbot contributed to the memorization of the relevant content by adding different senses to learning process. On the other hand, he defined SosyalciBot as “An educational material that supports teaching”. According to him, thanks to this material, they could go beyond classical lecturing. He said:

We have gone beyond classical lecturing with SosyalciBot… The stronger the material, the greater contribution it makes. For example, when I was in middle school, we used frog legs in a Science class. I don’t remember much about Science, but I remember that frog leg… SosyalciBot had a similar contribution to our lessons.

Student reactions through the eyes of the teacher

During the interviews, the teacher shared the reactions he had observed throughout the process. According to his observations, students enjoyed using the technology and had fun throughout the learning process. They were excited about using it and utilized the chatbot even at home.

Problems throughout the process

The teacher stated that the biggest problem he experienced was the lack of time. He emphasized that Social Studies classes are held for three hours per week, and the content is quite intense. Another issue, according to the teacher, was that SosyalciBot didn’t have a voice recognition system. The teacher mentioned that while the chatbot could send voice messages to students, they had to write messages to interact with it. According to him, This not only took time but also negatively affected the interaction.

4.5.2 Chatbots through the eyes of the teacher

The theme “4.5.2. Chatbots through the Assessment of the Teacher” encompasses the teacher’s evaluation of the chatbot used in the teaching process. The theme, along with its categories and codes, is presented in Fig. 9.

Fig. 9
figure 9

Teacher’s views in the scope of the theme “Chatbots through the Eyes of the Teacher”

Full size image

Positive features

The teacher made some positive interpretations of SosyalciBot. According to the teacher, one of the advantages of the chatbot was that it could be used outside of school. Another positive feature of the chatbot according to the teacher was that it was easy to use the technology. The teacher also appreciated the motivating speeches of the chatbot during Q&A activities. He said: “When the chatbot said ‘You are amazing!’ it increased the motivation of children… Another motivating expression was ‘Good job!’ These were so nice”. The teacher mentioned that, since the chatbot is an up-to-date technology, it attracted the attention of students. He emphasized that the feature of sending images was beneficial for making the acquired information concrete. Another positive aspect of SosyalciBot, according to the teacher, was the Q&A activities supported with instant feedback, clues, and different types of questions. These features, he believed, distinguished the technology from similar ones. The teacher frequently pointed out that SosyalciBot appealed to different senses, which he considered a positive attribute. Additionally, the social existence reflected by the chatbot, along with the story created to support this social presence, was welcomed by the teacher. He also appreciated the representation of the chatbot’s social presence in robot form, stating that this feature attracted the attention of the students.

Limitations and suggestions for improving the chatbot

According to the teacher, one of the most important limitations of the chatbot was that it did not have a voice recognition system. He mentioned that this was quite time-consuming, and making the chatbot voice-sensitive would increase its efficiency. Another limitation underlined by the teacher was stagnation. He suggested that the content provided by SosyalciBot should be enriched with videos and animations to make it more dynamic. The teacher also made alternative suggestions for improving the chatbot. The first is to transform the chatbot into a physical form. The teacher stated that chatbots in robot form would be interesting. The teacher also suggested that the chatbot be trained to teach different disciplines. Finally, the teacher stated that integrating humorous elements such as cartoons into the chatbot content would make the chat experience more interesting.

5 Results, discussion, and suggestions

In this section, the research findings were discussed within the framework of the existing literature, and suggestions were offered.

5.1 Results and discussion about the effect of chatbots on academic success

At the end of the experimental process, it was found that the post-test academic success of students in the experimental group was significantly higher than the success of students in Control 1, who received hybrid education, and Control 2, who received face-to-face education. When the literature is analyzed, a similar pattern emerges, indicating that the post-test academic success of experimental group students who received chatbot-supported learning-teaching processes was significantly higher than that of control group students (Essel et al., 2022; Hsu et al., 2021; Kim, 2018, 2019; Tham & Ruan, 2019; Vázquez Cano et al., 2021). In this regard, it can be said that this research finding aligns with the conclusions of various studies in the literature. Additionally, some study findings suggest that experimental group students who received chatbot-supported education exhibited higher academic success compared to control group students; however, this difference was not statistically significant (Grossmann et al., 2019; Yin et al., 2020). Nevertheless, the majority of the observed academic studies indicate that chatbots lead to a significant difference in post-test academic success among students. This finding suggests that the result obtained in this study is in line with the prevailing finding in the related literature.

The qualitative findings of the study, on the other hand, provide insights that can be considered explanatory factors for the academic success observed in the experimental group. Students stated that SosyalciBot presented the subject after the teacher and conducted Q&A activities. According to them, these were highly supportive as they not only had the chance to learn a subject from two different sources but also could reinforce their knowledge. The images presented by SosyalciBot were another factor that students emphasized since these images objectified what they learned during classes. On the other hand, students said that as the chatbot was used at the beginning of lessons, it increased their readiness, and they participated more in classes since they felt more prepared. In addition, since they could use the chatbot at home, they had the chance to learn a topic that they might have missed at school. Many students stated that the chatbot made the learning process more enjoyable and attractive, which positively affected their success and attitude toward the lessons. They mentioned that the language of the chatbot motivated them, while technology increased their interest. Additionally, functions such as Q&A activities, instant feedback, and clues contributed to the learning process. All these interpretations by students about the process and technology can be seen as essential factors that explain the success they achieved at the end of the process.

These positive views of students were also supported by the teacher who conducted the process. The teacher mentioned that students enjoyed the learning process thanks to SosyalciBot. They became excited while learning, making the process interesting. According to the teacher, students used the chatbot outside of school. SosyalciBot became a supportive tool in terms of reinforcing acquired information, readiness, motivation, and participation in classes. It can be said that all these qualitative findings are related to the significant difference observed in the experimental group at the end of the process.

5.2 Results and discussion about the effect of chatbots on permanent learning

At the end of the process, it was found that students in the experimental group were significantly more successful than those in the control group in terms of permanent learning. When the literature is analyzed, it is observed that there are not many studies indicating the relationship between permanent learning and chatbot technology. Tham and Ruan (2019) conducted an experimental research study to compare the effects of chatbots and flashcards on permanent learning. As a result of this study, it was determined that the group that learned with the chatbot had significantly higher scores than the group that learned with flashcards. At this point, it can be said that the result obtained by Tham and Ruan (2019) is similar to the finding of this study.

In semi-structured interviews, some students stated that the chatbot taught the subjects with images, presented the content with a combination of voice and text, enabled them to engage in clue-supported Q&A activities, and allowed them to use the technology at home. All these aspects were evaluated by them as factors contributing to the retention of information. They also mentioned that, as the chatbot covered the subject following the teacher’s instruction, they were able to reinforce their knowledge. The teacher, on the other hand, highlighted that the chatbot appealed to different senses, thereby enhancing permanent learning. All of these aspects can be viewed as explanations for the significant difference in the retention test scores of the experimental group students.

5.3 Result and discussion of the experiences of students about the use of chatbots in the process of learning social studies

Semi-structured interviews with students revealed different views about the process and the technology itself. Students stated that SosyalciBot could be used outside of school, giving them the chance to study on their own. Consequently, technology could contribute to autonomy in learning. In their qualitative study, Haristiani and Rifai (2021) analyzed the impact of a chatbot named Gengobot on learning Japanese grammar. Students who participated in the study mentioned that Gengobot improved their autonomy in learning; it was found to be beneficial and easy to use. Similarly, Jeon (2022) carried out a similar study, and the students stated that chatbots gave them the chance to enhance their autonomy in the learning process.

Students stated that SosyalciBot, unlike other technologies, can provide them with instant feedback. They also mentioned that technology makes learning interesting and is advantageous in that it is accessible 24/7. Similarly, Vazques Cano et al. (2021) determined that students in the experimental group had the chance to receive more feedback with chatbots than with the traditional method. According to the students, chatbots were easy to use and interesting, and they could always benefit from chatbots because it was possible to reach them anytime, anywhere.

An essential point often mentioned by students was the contribution of the Q&A module to their learning. Many students stated that it was positive to be able to receive clues and instant feedback in Q&A activities. Similarly, in the study by Pereira (2016), most of the participants made positive interpretations of making Q&A with chatbots, and they stated that chatbots could be used to test knowledge after classes in different subjects. On the other hand, it was revealed that carrying out Q&A activities with SosyalciBot was less stressful than answering the questions asked by real teachers. For instance, a student stated that he abstained from giving incorrect answers to his teacher in the classroom while he felt more comfortable in the Q&A activities of the chatbot. In the study by Jeon (2022), most of the participants mentioned that they felt less anxious when talking to chatbots, and they didn’t feel judged throughout the learning process. Fryer and Carpenter (2006) stated that students tend to feel more comfortable talking to chatbots when compared to real individuals.

The student frequently appreciated the language used by SosyalciBot; they mentioned that the language was kind, constructive, facilitating, and motivating. These factors made them feel relaxed and had positive effects on their psychology. Yang et al. (2021) pointed out the importance of considering human conditions while creating AI designs. They emphasized that while improving human intelligence with the support of AI, it is necessary to use a human-centered approach. At this point, it can be said that the language used by SosyalciBot is quite parallel to the views of Yang et al. (2021).

Another positive point mentioned by the students was that they felt as if the chatbot was a real human teacher. Greyling (2019) stated that humans had positive views about the anthropomorphic approach; it could thus be beneficial to make chatbots visually similar to humans and give human personalities to the chatbots. Although SosyalciBot wasn’t anthropomorphic in terms of shape, it can be said that it had a variety of anthropomorphic features such as congratulating, motivating, and transmitting emotions with emojis, which were evaluated positively by the students. In a similar study by Jeon (2022), students stated that they felt like talking to a real person while communicating with the chatbot. This feeling increased their motivation to use chatbots and activated their participation in speaking English. De Cicco et al. (2020) mentioned that the social presence, emotions, and empathy created by chatbots will have a positive impact on attitudes toward chatbots; this will eventually create an emotional connection between the chatbot and the individual. At this point, it can be argued that the positive attitudes of students toward SosyalciBot may be influenced by its social presence to a certain extent.

Participant students stated that the chatbot motivated them and increased their interest. Lee et al. (2011) found that chatbot-supported language training significantly increased students’ language learning motivation. Han (2020) determined that chatbots significantly increased students’ motivation, interest, and beliefs in learning English. On the other hand, their anxiety significantly decreased. The positive impact of chatbots on increasing student motivation was also revealed by the experimental studies carried out by Kim (2018) and Yin et al. (2020). Based on all this, it can be said that the positive views of students about the chatbot in terms of motivation and interest are parallel with a variety of quantitative research results in the literature.

The students also made suggestions for improving the chatbot, considering the limitations of the technology. For instance, some students mentioned that SosyalciBot can only be used when there is an internet connection, and turning the technology into an application could be beneficial as it would be easier to reach and use anytime. On the other hand, students suggested that videos and animations could be integrated into the chatbot to increase its dynamism. Additionally, different language options and enjoyable content could also be added to the chatbot. A limitation of SosyalciBot highlighted by students was the fact that it couldn’t answer all of the questions and used a limited language. As the experimental process of the study was under the control of the teacher, there was no problem in the communication between SosyalciBot and students; however, this limitation drew the attention of students. This situation is often observed in closed-domain and retrieval model chatbots designed around specific duties. These chatbots cannot interpret out-of-scope messages coming from students, and this finding was determined in many academic studies in the literature (Chuah & Kabilan, 2021; Jeon, 2022; Shin et al., 2021).

Similar problems are observed in studies related to the use of personal assistants, such as Siri, Google Assistant, and Alexa, which are advanced versions of chatbots, in language education. The students who participated in these studies stated that, although AI-powered personal assistants were supportive in the learning process, these assistants sometimes couldn’t understand them due to pronunciation differences. Furthermore, in some cases, the students found it challenging to comprehend the answers provided by the assistants (Dizon, 2017, 2020; Dizon & Tang, 2019; Moussalli & Cardoso, 2016; Underwood, 2017). All these studies indicate that AI-powered dialogue systems haven’t reached the desired level of development. However, chatbots have been improving every day with the advancements in AI, and it is foreseen that these programs will be used more frequently and effectively in educational environments in the near future (Kandpall et al. 2020).

5.4 Results and discussion about the experiences of practitioner teacher in terms of the use of chatbots throughout the process of social studies teaching

During and at the end of the process, interviews were conducted with a practitioner teacher regarding the process and chatbot technology. The practitioner teacher stated that throughout the experimental process, he could cooperate with the technology. He expressed that the technology proved to be valuable in enhancing student motivation, readiness level, and participation, and reinforced what is taught. According to the teacher, the chatbot activated different senses, leading to more permanent learning. He defined the chatbot as an easy-to-use and supportive technology that made the lessons more interesting and enjoyable. When the literature is analyzed, it is seen that the number of studies reflecting the use of chatbots from the viewpoint of teachers is limited. In the study conducted by Bii et al. (2018) most teachers stated that chatbots were useful, easy-to-use, beneficial, and interesting. The teachers emphasized that they enjoyed using chatbots as chatbots enlightened the contents of their lessons and contributed to understanding the topics. In their study, Chuah and Kabilan (2021) researched the viewpoints of 142 teachers who used chatbots for two months. At the end of the process, teachers stated that chatbots were supportive; they could correct the mistakes of students with feedbacks, and they were easy-to-use tools. At this point, it can be considered that the teacher’s relevant views on technology bear similarities to those of teachers in various studies in the literature.

The practitioner teacher stated that one of the positive features of SosyalciBot was that, although limited, it presented a social existence. The teacher mentioned that the Q&A process based on instant feedback and motivating sentences was similar to real teachers, and these features supported the social existence of SosyalciBot. Similarly, in the study by Chuah and Kabilan (2021), teachers stated that the chatbot developed for English education represented a social entity with its emojis, clues, and two-way interactive structure. On the other hand, the teacher appreciated the creation of a fictional story about the chatbot under the title “Who is SosyalciBot.” When the literature is analyzed, it is seen that creating stories about chatbots is a common trend. For instance, Kim et al. (2022) developed a chatbot called Ellie that teaches English. They described Ellie as a human who grew up in San Francisco, and Ellie was represented with an avatar that represents herself. Similarly, SosyalciBot was represented with an avatar, and the practitioner teacher appreciated this feature. There are studies in the literature about the contributions of chatbot avatars to the conversation and interaction processes (Angga et al., 2015; Tanaka et al., 2015). For instance, in the study conducted by Jenkins et al. (2007), the majority of participants expressed that having a chatbot avatar would make the conversation more interesting. Ciechanowski et al. (2019) found that chatbot avatars led to the impact of an “uncanny valley,” referring to the creepiness and discomfort often experienced in human-machine interactions. Böcker (2019) determined that avatars did not significantly affect the reliability and utility of chatbots. In this study, the practitioner teacher didn’t provide a detailed view of the use of avatars; however, he mentioned that using robot avatars instead of human avatars would be catchier.

The teacher mentioned that the most significant shortcomings of SosyalciBot were its lack of sound sensitivity and the absence of content such as videos and animations. The teacher suggested that SosyalciBot should be made sound-sensitive, and its content should be enriched. Another suggestion from the teacher was that the scope of SosyalciBot should be wider, and it should be implemented in other learning domains of Social Studies. When the rich content and different disciplines of Social Studies are taken into consideration, it can be said that this suggestion is quite applicable. Similar to the students’ viewpoints, the teacher welcomed the idea that the chatbot could contribute to memorization processes. It is important to note that although memorization isn’t preferred in educational processes, both students and the teacher welcomed this aspect. This fact may be related to the com petitive and rote-learning-based education system in Türkiye. In addition, the teacher stated that the class hours are insufficient to cover the intense content of Social Studies and emphasized that this situation created an essential problem in effectively using SosyalciBot in classes. All these results indicate that the weekly allocated 3-hour class duration for Social Studies in Türkiye is insufficient.

5.5 Suggestions

It is possible to make some suggestions, based on the theoretical framework of the research, findings, and researcher experiences obtained during the application process. The suggestions about the practice and future research are presented below.

5.5.1 Suggestions for practice

  • The research findings, along with the theoretical framework on the use of chatbots in education, indicate that chatbots have positive effects on academic success and permanent learning. At this point, Social Studies teachers can be recommended to explore the potential of chatbot technology and use them in their classes to ensure more efficient learning experiences.

  • In the qualitive dimension of the study, participants stated that the chatbot has some specific limitations and cannot go beyond the borders of a predetermined flow. This situation is common in closed domain and retrieval models designed around specific duties. At this point, it can be recommended to develop chatbots that are based on more advanced educational data and possess high-level communication skills. This requires an interdisciplinary viewpoint, developed machine learning techniques, the experiences of experts, time, and financial resources. At this point, it can be recommended that official institutions establish groups of experts to develop advanced chatbots capable of teaching specific subjects.

  • Another observation from the literature in this study is that the use of chatbots has primarily focused on language education. However, it should be considered that interactions between the user and the chatbot, such as “reading”, “writing”, and “listening”, are natural components of language acquisition. Social Studies involves numerous abstract concepts, and sometimes there might be more than one correct answer to a question. Moreover, it encompasses various intertwined disciplines, resulting in a complex nature. Considering all these factors, it can be said that achieving success with chatbots in Social Studies is relatively more challenging than in language education. At this point, it might be useful for Social Studies educators to incorporate multimedia content, such as images, video, and voice, more frequently in the chatbot development process. Additionally, it could be beneficial for them to integrate chatbots with pedagogy and content effectively and implement their well-planned lesson plans in alignment with chatbots.

  • As detailed under the heading of “3.5. Development of Chatbots”, the chatbot technology, the basis of this research, was attempted to be designed and employed following the constructivist paradigm. The researcher’s experiences suggest that the interactive nature of chatbots, which allows for two-way communication and instant feedback, can stimulate student-centered learning experiences. However, if not used appropriately, they can potentially promote a presentation-based teaching approach. At this point, it can be said that Social Studies educators should use chatbots according to a specific system that promotes active student participation.

5.5.2 Suggestions for future research

  • In the quantitative dimension of the research, the effect of chatbots was investigated concerning academic success and permanency. In light of this, it can be suggested to conduct quantitative studies to examine the effect of chatbots on various variables, including attitudes, motivation, and interest levels toward Social Studies.

  • The primary qualitative data collection method in this research involved semi-structured interviews. It can be suggested that future studies incorporate various data collection instruments, such as unstructured observations and student diaries, to provide a broader perspective on the technology and process.

  • In this study, the educational potential of chatbots was investigated in Social Studies education. However, the existing literature highlights a lack of awareness regarding the use of chatbots in fields such as geography, history, and economics education. It can be recommended to conduct new research in these areas.

  • One notable point in the research literature is the gap between chatbot technology and learning theories, underscoring the absence of a theoretical framework guiding educators in integrating chatbots with pedagogy. At this point, it can be recommended that social studies educators undertake theoretical or practical studies on integrating AI technologies, including chatbots, into constructivist learning environments.

  • This research incorporated specific ethical considerations in the process of developing chatbot technology. While literature reveals a growing interest in ethics and AI, there is a lack of studies exploring this relationship in Social Studies education. At this point, it is recommended that future studies explore the connection between ethics and AI in Social Studies.

  • According to the literature in this research, there is a high level of awareness regarding the use of AI technologies in education at the international level. However, relevant studies in Türkiye are still limited. It is recommended that researchers in Türkiye conduct studies aimed at integrating various types of AI technologies into educational environments. These studies are assumed to contribute to the domestic literature and enhance awareness of the use of AI technologies in education.