As a field interested in overcoming educational challenges with adaptive technologies, AIED has been strongly influenced by psychological research and findings. Many psychology theories have inspired and continue to influence the design and development of AIED systems. Consequently, the two following research questions are investigated in this section:
Do Detected Socio-Cultural Imbalances in Psychology Have Implications for the AIED Research Field?
Many AIED topics are strongly influenced by research in psychology, but it could be claimed that they are all about basic and universal human processes. To reject this hypothesis and confirm that cultural variations exist for psychology-related topics of interest to the AIED community, we first concentrate our investigation on ‘emotions,’ since this is currently one of the most popular research topics in AIED. There is indeed evidence that affective experiences vary across cultures in many ways.
Though universalisms in emotion have been identified (Ekman 1972),Footnote 5 cultural variations in various emotional features have also been extensively reported (e.g., Matsumoto and Ekman 1989). According to a literature review (Mesquita et al. 1997), numerous studies have identified cultural variations in affective antecedents (events or objects that trigger an emotion), subjective experiences (feelings), behavioral response, in physiological changes related to an affective experience, and in affective appraisal. Considering more specifically the last point, members of a cultural group commonly develop and internalize specific interpretations about a situation (Sharifian 2003). Hence the experience of a similar situation by culturally distinct peoples can result in different interpretations, and therefore in different affective reactions. For example, stereotypes are socio-cultural constructs that ‘simplify’ cognitive and behavioural reactions to situations (Jost and Hamilton 2005), and many of them have strong affective implications at different levels. Furthermore, the frequency in reporting positive or negative emotions is also known to vary across cultures, and cultural differences in processes pertaining to emotion recall have also been reported (Robinson and Clore 2002; Scollon et al. 2004). This implies that post-hoc assessments of affect may thus be culturally-marked. As a conclusion to their survey, Mesquita et al. claim that it is “convincingly demonstrated that there are cultural differences in the ecology of emotions” (p. 265).
Furthermore, research has identified cultural variations in educational practices and teachers’ beliefs (TALIS 2009), as well as in many other psychological topics of interest for the AIED research community, such as decision making, reasoning, motivation, self-regulation, collaboration, competition, personal values, and various aspects of perception and interpretation (Henrich et al. 2010a; Various authors 2010; Purdie and Hattie 1996; Lynch et al. 2009; Ryan et al. 2005; Hofstede 2008).
No evidence suggests that research on emotion and on other psychological topics of interest for AIED is informed by more culturally-aware practices. Hence it can be fairly assumed that psychological research that influences AIED is similarly WEIRD-imbalanced and that its cultural variability is mainly ignored.
Through transitivity, AIED research and activities are thus likely to be biased by this situation, which can manifest in many ways such as:
A lack of consideration of culture as a relevant factor to explain user responses to AIED systems. This is especially important when collecting data to inform human features that are known to be culturally sensitive such as self-concepts, motivation, or emotion.
A lack of interest in or an ignorance of educational issues in non-WEIRD societies that could benefit from AIED paradigms.
The dominance of system and interaction designs that may be appropriate for WEIRD audiences but less optimal for other populations. This could actually result in additional challenges for non-WEIRD scholars wishing to integrate with the AIED community. It may be more difficult for these scholars to obtain similarly effective systems when grounding their research and developments on paradigms that are possibly less natural and appropriate for their local environment. This can therefore result in more negative results and less successful systems, which are harder to publish.
Overall, problems resulting from cultural imbalances in psychology are likely to similarly apply to the AIED research field (see the previous section on implications of the detected USA and WEIRD imbalances). However, there might be a balancing effect to this bias. AIED is a technological research field and it penetrates societies as information technologies do. Given the existence of a Digital Divide phenomenon, AIED systems that have gone beyond laboratory testing in the last few decades have most often been deployed in WEIRD environments and especially in US institutions and organizations. Consequently, biasing effects induced by socio-cultural imbalances in psychology might have been somewhat limited since typical AIED end-users would be mostly similar to typical subjects of psychology research so far.
Nevertheless, and as reported by Henrich et al. (2010a), socio-cultural variations exist and are identified between (and within) the different WEIRD societies as well as between typical psychology subjects and the rest of the WEIRD population. Despite AIED end-users being mainly from WEIRD countries and especially from the USA, psychological findings obtained from specific populations may still have influenced AIED systems targeting other populations even though the appropriateness of such a generalization may not have been established at all.
As a summary, it can be fairly assumed that socio-cultural imbalances detected in psychology have a biasing effect on past and present AIED research activities, but these negative influences might have been limited by the fact that typical AIED end-users could have been most of the time US and more generally WEIRD individuals so far.
Is AIED Inclined to Produce WEIRD-Imbalanced Research?
The fact that issues in psychology spread to and influence AIED activities is relatively obvious considering that psychology is historically important for this research field. However, the AIED community is not only a consumer of external research findings, it also produces its own studies, norms, practices, and theories. This section thus investigates whether AIED is affected by similar socio-cultural imbalances to those detected in psychology, and is likely to produce culturally-biased research.
The methodology employed here was largely inspired by the one described in Arnett (2008), and consisted of analyzing AIED research. Consequently, full paper publications in two AIED-related international conferences were analyzed over a period of 12 years (2002–2013). Selected conferences were the International Conference on Intelligent Tutoring Systems (ITS) and the International Conference on Artificial Intelligence in Education.
A conference-focused analysis was preferred to the journal-based approach of Arnett because the AIED community includes scholars from several different disciplines. Therefore journal publications dedicated to AIED research are spread not only among the few journals dedicated to the field but also among many others that are discipline-specific.
AIED and ITS conferences were chosen because their content is expected to correctly reflect state-of-the-art AIED research production.Footnote 6 They are commonly acknowledged as the two top-tier conferences for AIED research, and are perceived as major yearly events for the AIED community. Although other conferences of quality exist (e.g., ICCE, ECTEL, ICALT), it is commonly agreed that no other event has achieved a similar level of international recognition in the AIED field.Footnote 7
Both selected conferences are biennial: ITS2002 occurred in France and Spain, AIED2003 in the USA, ITS2004 in Brazil, AIED2005 in the Netherlands, ITS2006 in Taiwan, AIED2007 in the USA, ITS2008 in Canada, AIED2009 in UK, ITS2010 in the USA, AIED2011 in New Zealand, ITS2012Footnote 8 in Greece, and AIED2013 in the USA.
Origins were coded according to the following categories: USA and English countries (Canada, Australia, New Zealand, UK, and Ireland) were considered separately as in Arnett (2008) whereas other categories were related to continents: Europe, Asia,Footnote 9 and Africa.Footnote 10 A Latin America category was also considered and was essentially composed of entries from South America and Mexico.Footnote 11
National affiliations of first authors, of other authors, and national origins of samples were recorded, and the rest of the coding approach was similar to Arnett (2008). If an author had several affiliations in the same category or if a paper included several samples from the same origin, only one entry was recorded. However, if an author had institutional affiliations in multiple categories or if a paper included several samples related to different categories, one entry was recorded for each related categories. This occurred on extremely rare occasions.
The following sections present records of international representations in AIED authorship and AIED samples.
International Representations in AIED Authorship
The following tables present national affiliations of first authors (Table 4) and other authors (Table 5) in the twelve last ITS and AIED conferences.Footnote 12
Results reported in Tables 4 and 5 illustrate very strong WEIRD-imbalances in authorship, although most of the time these are slightly lower than those identified in psychology (Tables 1 and 2). First authors and other authors are affiliated to US and WEIRD institutions to a very large extent. Latin American and Asian authors have extremely low representations and, furthermore, almost all of them come from very limited subsets of countries: essentially Brazil and Mexico for Latin America, and Japan, South Korea, Taiwan, and Singapore for Asia. Continental China and India have almost no author representation even although they together account for one third of the World population and have a growing importance in the global economy. No first authors from African institutions are reported and very few other authors that all come from the Maghreb or Egypt.
International Representations in AIED Samples
Because of the interdisciplinary nature of the field, some papers published in ITS and AIED conferences do not include human-based evaluations whereas other papers use human-based samples and datasets exclusively to validate technical features such as machine-learning or clustering algorithms. Only papers with evaluations that did (at least partially) reflect upon human features were considered. Datasets could be considered as long as data collection was appropriate for socio-cultural investigations, even if they had not been collected for a specific study and rather came from a data repository. Evaluations based on simulated students were discarded because they model only a limited set of characteristics of human beings, and there is no demonstration they exactly mimic complex aspects of human students such as cultural features.
Figure 1 presents the percentage of papers with either human-based evaluations or investigations of human features in the twelve considered conferences. It clearly shows that the frequency of papers respectively with human-based evaluations and investigations of human features have both notably increased in recent conferences.
The goal of the next part of the analysis was to report origins of AIED samples of interest, i.e., those used to investigate human features. However a significant share of these samples did not include clear information about the country of origin but, in most cases, it was possible to safely infer their origins by cross-checking indirect clues. Nevertheless, a few additional samples were discarded because of the impossibility of determining their origin with sufficient confidence. Table 6 presents the national origin of the remaining samples of interest from the 12 ITS and AIED conferences.
Results in Table 6 indicate US and WEIRD imbalances similar to those reported in psychology (Table 3) although the “English countries” cluster accounts for a greater share of samples. Asia is slightly more represented in relevant AIED samples than in psychology ones. There are almost no relevant South American samples, and absolutely none from Africa.
Discussion and Analysis by a Panel of AIED Experts
Methodology and Panel Presentation
In order to assess these results in a non-dogmatic way, several scholars with strong AIED expertiseFootnote 13 were contacted. A first group of 7 experts whose comments were mentioned in an earlier-yet-limited version of this work (Blanchard 2012a) was later completed by 5 additional experts.Footnote 14 They were all presented with the same information in the exact same way. It consisted of a word document with a shorter overview of Arnett (2008) and Henrich et al. (2010a), followed by data collected from ITS2002 to AIED2011, inclusively, since data for ITS2012 and AIED2013 became available after the release of this first paper version. The document was followed by an open-ended questionnaire to assess a) how panel members analyzed the data, and b) whether they agreed to some suggestions for addressing the situation.Footnote 15
In this panel of 12 experts, there is an equal number of men and women. Regarding their academic affiliation, 3 are from the USA, 4 are from English countries, 3 are from Europe, and 2 are from Asia. Regarding their academic background, 3 report a PhD in AIED/educational technologies, 2 in psychology, 3 in computer science, 2 in human-computer interaction, and 1 in artificial intelligence. Since the objective was to obtain feedback from the current AIED community, it was important to have a representative panel. Origins of affiliations seem to fairly represent typical attendance of ITS and AIED conferences even if, despite numerous attempts, no targeted South American experts positively replied to the invitation. No expert with an African affiliation and meeting the AIED expertise criteria could be identified. Finally, the composition of the general community evolves over time and there is no definitive information about discipline representation within the conferences. However, efforts have been made to select experts to represent both the technical, social, and humanities dimensions of the field.
The following section summarizes views and comments of panel members that were collected individually. Specific care has been put into fairly illustrating as many of the expressed opinions as possible, which means that some of these positions are not necessarily shared by other panel members or the author.
All experts acknowledged that the collected data demonstrate strong USA and WEIRD dominance in AIED research. For most of the experts (and for the author as well), it was important to insist that the detected imbalances are unintentional, and that the selection of papers is only based on scientific criteria.
All experts agreed to different extents that these imbalances are very likely to influence AIED activities, although some of them rightfully insisted that no direct results are actually provided on how these imbalances may be impacting AIED, and consider that it may be too early to use the word ‘bias.’ Several panel members considered the AIED field to have several important differences when compared to psychology, and believed that the imbalances would have implications that are unique to this community. They suggested therefore that investigations to identify these effects should be undertaken: more master and doctoral students should start projects to investigate how culture and AIED intersect, and more AIED-related events should include cultural considerations in the future.
Several experts from different origins reported the same following position: a US bias in AIED is not surprising because an American-centric view of the World is affecting many other areas of daily life in many other societies and, according to one expert, there is a worrying “strong tendency to blindness to that bias.” Although the influence of the US culture on other societies is likely to be the strongest, several other countries are also culturally influencing other societies (e.g., France vs. several French-speaking countries).
Another expert noticed that AIED research is already reaching various socio-cultural groups, within the USA especially, according to ethnic, economic, and localization criteria, and consequently claimed that “AIED is doing better at involving diverse populations than [the reported data] implies.” Several other experts further insisted that beside socio-cultural representations, other potential sampling imbalances in AIED should be investigated as well.
Panel members were also asked to reflect on several actions to potentially mitigate the impact of detected socio-cultural imbalances. Most experts agreed that in order to achieve more culturally-aware AIED research, the AIED community needs to be made aware of the situation as a necessary first step. This was indeed the main objective of this paper. Positive improvements should first and foremost come from community members better at self-regulating their research efforts, e.g., by including socio-cultural considerations more frequently in their research design and analyses. Likewise, experts agreed that there is place for improving sample descriptions. For example, authors should more systematically describe appropriate contextual factors such as nationality, socioeconomic status, ethnicity, gender, etc.Footnote 16 whereas reviewers could more frequently highlight ill-presented samples. The considered population should also be more commonly highlighted in paper conclusions.
Several experts agreed that organizing conferences more frequently in non-WEIRD societies would help raise the diversity of the community. Indeed, it is to be noticed that of the 12 considered conferences, only 2 occurred in non-WEIRD societies: ITS2004 in Brazil and ITS2006 in Taiwan, and these conferences respectively had the highest rate of papers from Latin America and Asia (see Table 4). More generally, conference localization almost always impacted positively on the representation of researchers from surrounding countries. However, another expert insisted that conferences should mainly be organized in countries that are easy to reach by researchers from established AIED countries. Otherwise the overall quality, attendance,Footnote 17 and consequently the long term credibility of conferences could be threatened. This position is not necessarily in opposition with a more frequent localization of conferences in non-WEIRD countries. There are many intertwined factors that influence the decision of organizing a conference in a specific location. A decisive one is that WEIRD scholars may be more experienced at submitting conference organization proposals which are consequently more successful.
Many panel members suggested that the AIED community should question itself about research it should focus on developing. Several experts reported that investigating socio-cultural variations (or universalisms) in different contexts is not perceived as bringing ‘new’ findings by some reviewers. They invite the community to reflect upon this point. Another expert posited that reported cultural imbalances would not be an issue if the AIED community correctly followed the ‘scientific paradigm,’ which (s)he claimed is not currently the case.
Finally, several panel members with different academic profiles also criticized the current importance of human-based evaluations on paper acceptance/rejection decisions although they do not question their general interest for the field. Some of them consider that an evaluation, even if loosely done, now has too much impact on paper acceptance. They see this situation as very problematic since a paper only detailing a clever technical solution might sometimes be a better contribution to the field although it seems harder to get published. Continuing into this direction, one of the experts with a background in humanities argued that over the past decade, AIED has been distracted from its innate quest of investigating “how to automate interaction in a teaching/learning environment,” or how to “develop representations and algorithms by means of which [educational] software can adapt their interaction.” Instead there are more and more studies attempting to investigate “truly psychological issues related to education” that (s)he claims should remain the prerogative of other disciplines “such as Developmental Psychology, etc.” Other panel members have produced similarly virulent critiques based on a perception of a growing importance for human evaluations in the paper selection process.Footnote 18 A panel member also reported the case of an AIED senior member focusing on this point when commenting on Blanchard (2012a) during an academic event.
This section provides additional culture-related insights that result from several years spent investigating cultural disciplines and theories, and from interactions with various scholars interested in the emerging interdisciplinary field of culturally-aware computing.
Non-WEIRD Societies as Emerging AIED Markets with High Potential
As it has been stated, negative effects of identified socio-cultural imbalances in samples might have been limited by the fact that typical past and current AIED end-users might mainly be WEIRD and US individuals, the very cultural clusters that have been mostly investigated. However, the Digital Divide phenomenon is rapidly fading in the twenty first century. IT technologies are becoming more ubiquitous in both developed and developing non-WEIRD countries, and this could bolster the global dissemination of AIED systems.
Nevertheless, the needs of WEIRD societies for AIED technologies remain and so does the need to keep addressing them. The main objective in these contexts is to improve established educational systems and opportunities. However, in non-WEIRD societies, challenges that AIED will have to tackle could be slightly shifted. Of course, AIED should contribute to the improvement of existing educational systems of non-WEIRD countries as well. But many of these societies have been and keep experiencing massive societal changes. With the rapid growth of middle-classes in demographic giants such as China, India, Brazil, the Philippines, or even Nigeria to name but a few, the most important challenge for AIED may thus rather consist of offering educational opportunities that are currently missing, and addressing the lack of efficient educators in many disciplines and at different levels.
These are great opportunities for the AIED community to have a positive life-changing impact on many human beings, and for AIED systems to reach new economically-viable markets. The way the AIED community decides to tackle this challenge will have its importance. It may choose to consider that these new audiences do not differ from those already researched, and thereby adopt an approach that is close to neo-imperialism, or it may develop and follow guidelines and practices to better consider and respect cultural diversity and specificities (see for example UNESCO 2007).
Developing AIED Investigations in Foreign Cultures
More studies should investigate AIED challenges on samples representing different populations than those of the USA and other WEIRD countries. However, there is a lack of experienced AIED researchers in many non-WEIRD countries and consequently investigations on the related populations will largely depend on initiatives of WEIRD scholars in the short term. Setting up an evaluation on a sample from a distinct culture is not a trivial task and may provide data that seems to initially answer a target question although they actually inform other features.
For example, testing a system or a methodology on other culturally-distinct populations does not necessarily provide information about its universality. A population from a country Y may be commonly acquainted with aspects of the culture of a country X. When testing if a system S developed to suit X is correctly used by population Y or if it produces similar results, one rather investigates features related to the cultural intelligence (Earley and Mosakowski 2004) of individuals from the Y population, that is, whether their knowledge of the X culture allows them to understand and interact with S. A successful usage of S by population Y, or the report of positive emotions in this situation, are both interesting results and could inform several research questions, but without further investigation they neither demonstrate that S is similarly adequate or optimal in countries X and Y, nor that some of the features of S do not seem odd to population Y. Analyses that would investigate whether WEIRD-produced tools and questionnaires can be used or answered by a distinct population rather than the original one they were conceived for would also be more likely to assess cultural intelligence features.
As it has been stated earlier in this paper, different cultures lead groups to endorse different dominant ecologies of conceptualization (see Sharifian 2003). Studies on culturally-specific conceptual frameworks related to education at large have been relatively limited so far. Hence, exploratory investigations that are open-ended and that do not focus on assessing WEIRD-defined constructs are greatly needed and should open many new and exciting research trends for the AIED community. As suggested by Gosling et al. (Various authors 2010, pp. 94–95), increasing global access to the Internet could facilitate such data collection, but the AIED community could also propose and develop dedicated technologies to reach diverse human populations, and procedures for controlling data collection.
A particular challenge is to design and produce systems dedicated for a target population. It may be essential for researchers to first develop a thorough expertise of the target population such as through reading ethnographic, anthropological or sociological surveys. However, and as it has been stated before, the cultural variance of features is frequently hidden and subconscious, and it may be impossible to achieve a high level of confidence about the absence of cultural oddness unless proper cultural representatives are involved.
Mentoring Programs to Increase Diversity
As discussed previously, investigations led by US and WEIRD AIED scholars naturally insist more on research topics of interest in US and WEIRD contexts (see Various authors 2010, pp. 84–85). However, many non-WEIRD societies are emerging markets for the AIED research field, as explained previously. These cultural contexts present education-related challenges that are totally absent from US and WEIRD societies, although these issues may be critical for raising the interest of non-WEIRD populations toward AIED technologies and facilitating their adoption in new contexts. There is no better way to identify these hidden needs than to have more AIED members from underrepresented societies. More diversity within the AIED community could also result in the emergence of alternative solutions to those typically proposed by WEIRD researchers.
However, potential community members may not be accustomed to the specific culture of the AIED community or to various challenges pertaining to the peculiar task of writing solid scientific papers. Making the same observation for psychology, Arnett suggested reviving a program where experienced authors develop a mentorship relation with community outsiders and/or newcomers. These mentors can suggest steps to improve the quality of research production to specific community standards. Developing a similar program in conjunction to AIED-related events has been similarly suggested by various members of the community.
Evolution of the AIED Culture
The description of an increasing share of conference papers using human evaluations shown in Fig. 1, as well as the related comments in the last paragraph of the panel section, both show that the culture of the AIED community has evolved in the last decade. Given the current rate of human-based evaluations in accepted full papers (generally close to or over 90 %), the inclusion of such a feature in paper submissions can be considered a de facto norm that emerged over time. This is a typical example of a cultural evolution process (see Henrich and McElreath 2007, for more details). This evolution is not consensual, however, and comments of some panel members illustrate that frustration exists toward it within the AIED community.Footnote 19 Detractors of this evolution especially argue that it provides unfair advantages to people with expertise in humanities who are trained for developing and presenting human-based evaluations. They also suggest that many technological research areas have weaker chances of appearing in proceedings given the importance that some reviewers give to human-based evaluations. Promoters on the other side insist on the human dimension of AIED and reason that demonstrating the effectiveness of learner-centered technological solutions without these evaluations is impossible.
This paper cannot settle the argument on whether this evolution is for the greater good of the domain. Nevertheless, from a cultural standpoint, this state of affairs presents risks quite similar to others that have been discussed in this paper. Researchers with expertise in humanities, being overall more skilled at preparing and presenting human-based results, may encourage greater representation of humanities research topics to the detriment of other topics that are conveyed by scholars with a more technological profile. This is particularly relevant for scholars from countries where technological curricula do not commonly include training in disciplines such as social science statistics, or where academic systems and institutions make interdisciplinary teams harder to establish.
Options for addressing this situation may also be relatively similar to those related to international representations in psychology or AIED. The community could consider that this norm is adequate, that detractors are a minority, and just ignore their position. It could also still consider that the norm is adequate but acknowledge that it disadvantages a creative part of the community and the specific research questions it conveys. To mitigate this effect, mentoring strategies could be developed to support researchers less skilled at developing and presenting human-based evaluations. Another option for the community would be to consider the norm as inadequate and propose measures to control the importance of human-based evaluations. For example, conference tracks targeting humanities-related papers, technology-related papers, or hybrid papers could be proposed and implement different paper selection approaches.