Socio-Cultural Imbalances in AIED Research: Investigations, Implications and Opportunities

Abstract

This paper investigates international representations in the Artificial Intelligence in Education (AIED) research field. Its methodological and theoretical groundings are inspired by Arnett (2008) and Henrich et al. (2010a) who addressed the same issue in psychology, and respectively a) discovered massive imbalances in representation in top-tier psychology journals, and b) clarified risks of this situation. Data on research production collected on 2 top-tiers AIED conferences indicate that relatively similar imbalances exist in AIED. Potential threats and challenges induced by that situation are discussed as well as additionally identified phenomena related to the culture of the AIED community.

Introduction

The Artificial Intelligence in Education (AIED) research field aims to develop technologies for improving existing educational knowledge and practices, creating new ones, and/or overcoming a lack of teaching opportunities and specialists. Findings from multiple research disciplines are merged to achieve these objectives, with computer science and psychology being the historical cornerstones of this interdisciplinary research field.

As with any other scientific domain, AIED has to rely on theoretical foundations, which are most often provided by psychology and related domains in the humanities. Indeed, since its emergence as an individual research field, psychology has suggested educational problems to be addressed by AIED researchers. It has guided these scholars in defining novel education-related use of computer science techniques, or in proposing original approaches. Consequently, psychology currently has a key influence on many features of AIED systems such as their design, internal mechanisms, graphical user interfaces, and expected human-computer interactions.

In 2008, Jeffrey Arnett published an influential survey paper in American Psychologist (Arnett 2008) whose objective was to investigate international representations in contemporary psychology research production. Data that Arnett collected demonstrated that US and, more generally, Western, Educated, Industrialized, Rich, and Democratic (WEIRD)¹ societies have an extreme dominance on psychology content production, both in terms of authorship and sample representations. Arnett concluded his paper by stating that the reported dominance is so high it could negatively affect psychology as a human science.

Influenced by the work of Arnett and other similar studies, Joseph Henrich and colleagues later released an extensive interdisciplinary review to determine whether typical psychology samples are related to populations that are similar enough to the rest of humankind for generalization (Henrich et al. 2010a, b). Although the survey identifies evidence of universalisms, it also reports numerous studies that describe cognitive and behavioral socio-cultural variations between the limited populations used in psychology samples and other human groups.

Inspired by the studies of Arnett and Henrich et al., the objectives of this paper are threefold:

• to discuss potential implications for AIED of socio-cultural imbalances detected in psychology,

• to assess whether international representations in AIED are similarly imbalanced, and if so,

• to discuss potential implications of AIED imbalances and strategies to address them.

The next section describes prior investigations of international representations in psychology and more specifically the work of Arnett (2008) and Henrich et al. (2010a). Then, a similar investigation of international representations in AIED is described, followed by commentaries collected from an international panel of AIED researchers and additional notes from the author of this paper.

Discussing International Representations in Psychology

This section is organized around the presentation of two recent yet influential papers: a survey published in American Psychologist that investigated international representations in contemporary psychology research production (Arnett 2008), and an interdisciplinary review that considered whether WEIRD populations are similar to others in their cognitive and behavioral functioning (Henrich et al. 2010a, b).

Quantifying International Representations in Psychology

The main thrust of Arnett’s paper (2008) consists of a systematic analysis of publications in six flagship APA journals seen as top-tier publications in their respective psychology areas by scholars around the world. These journals are Developmental Psychology (DP), Journal of Personality and Social Psychology (JPSP), Journal of Abnormal Psychology (JAP), Journal of Family Psychology (JFP), Health Psychology (HP), and Journal of Educational Psychology (JEP).

Arnett recorded national institutional affiliations of first authors, other authors, and samples according to the following regions: Africa, Asia, Europe, Latin America, and Middle East. He also grouped the United Kingdom, Canada, Australia, and New Zealand into a specific category (English-speaking countries) for historical and cultural reasons. USA was a category on its own since it was the main focus of his analysis, and Israel was also considered separately because Arnett assumed it did not fit well in other categories. Tables 1, 2, and 3 present Arnett’s results.²

Table 1 National affiliations of first authors in five APA journals, 2003-2007

Table 2 National affiliations of other authors in five APA journals, 2003-2007

Table 3 National affiliations of samples in five APA journals, 2003-2007

Strong imbalances are observed for each of the investigated features: 73 % of first authors, 74 % of other authors, and 68 % of samples are related to the USA. Broadening the analysis, results increase sharply to 97 % of first authors, 98 % of other authors, and 95 % of samples related to WEIRD countries, an extremely high representation rate for countries that accounted for only 5 % (USA) and 12 % (WEIRD) of the World population at the time of the study.

As a comparison, Latin America and Africa are almost absent from the records, and Asia is extremely underrepresented when considering both its demographic and economic weights in the World. Additional analyses by Arnett on productions for years 1988, 1993, and 1998 confirmed that this assessment can be extended not only to the 2003–2007 period but at least to the past 20 years.

Furthermore, when looking closer at samples with US origins for the year 2007, Arnett also noticed that many socio-cultural groups were underrepresented and that, in the specific case of JPSP, 67 % of US samples consisted of undergraduate psychology students. This social imbalance was even more salient in non-American studies published in JPSP where undergraduate psychology students constituted 80 % of the samples. This further supports Arnett’s position that a very narrow portion of humankind exercises particular influence on research in psychology.

Several factors can help to explaining these detected imbalances. Some of the main ones stated by Arnett are discussed in the next section.

Possible Explanations for the Detected Imbalances

Economic differences are certainly important factors relevant to this state of affairs. Many non-WEIRD societies are in developing countries where psychology research is far less funded than in WEIRD countries. However, this argument does not explain the extremely low representation of rich and research-intensive countries such as Japan or South Korea.

The status of English as the ‘lingua franca’ in contemporary research definitely facilitates production by authors with English as their mother tongue. Although many authors from non-WEIRD countries have sufficient English proficiency and frequently publish in this language in local or international journals, they are rarely able to ‘score’ top-level publications and international tribunes that the selected APA journals represent. Consequently, the findings of these authors are less widely disseminated, less taken into consideration, and remain largely ignored by American and WEIRD psychologists (Denmark 1998).

Arnett suggests that the dominant philosophy of science in psychology especially nurtures the detected imbalances. According to him and others (e.g., Norenzayan and Heine 2005), American and European psychology has evolved since the 19th century with the overarching goal of controlling “the experiment so that distracting variables of real life could be stripped away in order to reveal the essence of a phenomenon” (p. 610), without considering how different participants are in their life outside of the laboratory. Consequently, Arnett and others describe the dominant philosophy of science in psychology as focusing on the investigation of “fundamental processes, resting on the assumption – rarely stated, and rarely actually tested – that people anywhere can be taken to represent people everywhere, and that the cultural context of their lives can be safely ignored” (p. 610).

Inspired by Cahan and White (1992), among others, Arnett insists that a “second psychology” has always existed and had numerous proponents such as Dewey, Vygotsky, and Wundt. Its contemporary expression, cultural psychology (see Heine 2012), has indeed emerged in the past 15 years as a respected research area that studies the cultural basis of human psychological characteristics. However, according to Arnett (2008), its influence remains limited since psychology studies commonly avoid investigating socio-cultural factors.

Other explanatory factors can be proposed and identified. Nevertheless, understanding precisely the reasons for this state of affairs will not make these imbalances disappear, although it may suggest actions in order to mitigate their magnitude. The main point is rather figuring out whether these imbalances are problematic and threaten psychological findings and theories. In order to provide a fair answer to this question, it is important to first discuss potential criticisms and limitations of Arnett’s analysis.

On Potential Criticisms and Limitations of Arnett’s Work

Arnett based his analysis of authors’ international representations on the origin of the authors’ institutions, not on the individuals themselves. Authors of non-WEIRD origins have certainly been appointed in American and WEIRD institutions. Similarly, authors originating from the USA and other WEIRD societies are affiliated with non-WEIRD institutions and their work may have also been successfully published. Overall, childhood, education, local academic cultures, and daily social experiences are some of the many features that may shape and influence research practices, interests, and reflections of scholars. Approaches that use only a few of these features tend to limit the inferences that can be made regarding an author’s cultural profile. Reporting on the national affiliations of institutions is not perfect, however it has the advantage of being the only feature that can be objectively gathered from paper analyses for indirectly informing about the cultural profiles of authors. In any case, this limitation does not apply to imbalances reported for samples.

Investigating socio-cultural variations through the analysis of data related to countries or country clusters surely presents limitations. There are indeed massive socio-cultural variations within countries and country clusters. On the other side, cultural proximity exists between countries that are categorized in different clusters. However, research has long established that cultural similarities can be identified between citizens of a country as well as between populations of different countries within specific country clusters (see, e.g., Hofstede et al. 2010; House et al. 2004; Livermore 2013). The level of granularity chosen by Arnett is thus commonly considered as valid for investigating socio-cultural trends. It is actually one of the most common approaches in cultural studies. However, readers have to remain careful and not assume that tendencies identified for a large entity similarly affect all its component members (e.g., country clusters versus countries, or countries versus citizens or sub-populations), an oversimplification commonly found in research (Blanchard 2012b).

Another potential confusion may lie in the use of journals of the American Psychology Association to investigate American and WEIRD influences on psychology content production. Because these journals are edited by an American association, wouldn’t it be normal that they essentially have American content? However, this is a false assumption, and one must not confuse an authority (i.e. APA) ensuring scientific credibility with the content it focuses on. APA journals investigate psychology under the perspective of a Human science and not under the far narrower view of a science about American people only.

Finally, Arnett performed his analysis of the domain by investigating a rather limited set of selected journals. It should first be noted that a similar approach has been used to represent various areas of psychology in past studies (e.g., Robins et al. 1999; Gannon et al. 1992; Graham 1992). The selected APA journals are not just any journals. They are top-tier ones and have all established strong international credibility and reputations in their respective psychology area, inspiring standards and influencing trends for the whole discipline. They claim the benefits of being regarded as top-tier journals not only by American scholars but by foreign ones as well, and even though other quality psychology journals exist, few have achieved a similar level of international recognition and leadership (and almost all are based in either the USA or an Anglo-Saxon country³).

Though previously discussed criticisms and limitations call for careful interpretations of the results, it would be very ill-advised to reject the existence of USA and WEIRD imbalances in psychology, especially when considering the fact that several other surveys have reported similar imbalances (e.g., Cole 2006; Denmark 1998). The next section discusses implications of these imbalances while commenting on a few additional hypotheses.

Implications of the Detected USA and WEIRD Imbalances

Should readers be convinced of the existence of strong USA and WEIRD-imbalances in psychology production despite the limitations previously discussed, it then becomes interesting to investigate whether these imbalances are problematic. In order to provide an appropriate answer to this question, socio-cultural imbalances in authorship and samples are discussed separately in the next sections.

Cultural Imbalances in Authorship

Researchers from various cultural sciences have long agreed that human beings are frequently unaware when their thoughts, practices, interests, etc. are culturally-specific (i.e., when they are more ‘normal’ in their socio-cultural environment than in others) (see e.g. Hofstede et al. 2010; House et al. 2004). This commonly leads individuals to miss or ignore the importance of thoughts, practices, interests, etc. that are not typical for them, although they may well be the expression of others’ normality.

It would be an arrogant assumption to expect scholars to be immune to this effect, and the detected imbalances in authorship are very likely to influence the list of issues deemed of interest by the psychology research community. Psychology may be overemphasizing issues that are common to WEIRD people (but not necessary to non-WEIRD people), and similarly neglecting those that are uncommon or nonexistent in WEIRD societies while being relevant in others. Likewise, WEIRD perspectives and solutions to psychological issues are more likely to be investigated than non-WEIRD ones.

Performing additional analyses on research production from 2007, Arnett also found that WEIRD scholars were not only dominating authorship, they were almost exclusively the associate editors, editorial board members, and consulting editors in all of the selected journals. This additional imbalance may well be a consequence of the dominance in authorship (efficient and influential scholars are those who are typically offered these roles), but it may also be another possible cause for the detected imbalances: overtly WEIRD-dominated editorial boards may introduce a bias by unconsciously considering WEIRD psychological issues to be of more interest than non-WEIRD ones.

The overrepresentation of WEIRD authors also has implications on the design of experiments. According to Baumard and Sperber (Various authors 2010, pp. 84–85), scholars embed their own ecology of norms, interpretations, and procedures into their experiments, but samples from different origins could adopt other interpretations that are more natural to them and consequently react differently than initially expected by scholars. It is thus challenging and not straightforward to replicate experiments on populations that have not been planned for, and no assumption should be made on the universal interpretation of procedures.

Instead, adapting experiments by thoroughly considering local socio-cultural conceptualizations is essential in most cases for ensuring that the initially targeted psychological feature remains the one being informed, and that no external bias is introduced. The involvement of non-WEIRD scholars in such reformatting processes could be key in appropriately reproducing WEIRD evaluations in non-WEIRD contexts.

Cultural Imbalances of Samples

Cultural imbalance of samples may carry larger risks. However conditions exist where they would not necessarily be an issue.

• Results that have been collected may be applicable solely to people similar to those that have been investigated, i.e., on WEIRD people only if the broadest approach is considered in the current case. In many situations, this is a totally acceptable consideration since research on psychology definitely has implications in, and can be beneficial to WEIRD societies. But Arnett and others (Sue 1999; Henrich et al. 2010a) have noticed that many scholars commonly extend the applicability of their findings to humans in general either by not stressing clear restrictions about the validity of their findings to a target population or by more directly claiming a generalization to other groups.

  The fact that researchers do not clearly limit their findings or that they claim a universalism without proper care actually leads to a similar result; psychological findings are commonly extended to populations that they have not been tested on. Should these generalized visions become attractive and popular enough among scholars to the point of being extensively cited, they may even gain the status of a de facto expression of ‘truth’ for these untested populations, and contaminate other studies that would refer to them in their theoretical groundings.

• American and WEIRD populations might be similar enough to other societies in their psychological and behavioral functioning so that it would be harmless to generalize findings obtained on WEIRD samples to other populations. This is certainly the case in several situations since universalisms exist. Danks and Rose (Various authors 2010, pp. 90–91) indeed argue that uniformity exists in “learning processes (broadly construed),” and most researchers agree that many internal processes of different kinds exist at the species level and are common to all human beings. However, this position does not conflict with the view that in many cases, outputs of these processes are directly dependent on external features including socio-cultural ones (see for examples appraisal theories of emotion, e.g., Lazarus 1991; Scherer 2005).

  A seminal review paper that reports numerous indications of cultural variations between WEIRD populations and others was released in 2010 (Henrich et al. 2010a, b). It is summarized in the next section.

Is it Appropriate to Generalize Imbalanced Findings of Psychology to Humankind?

Socio-Cultural Variations in Cognitive and Behavioral Functioning

Henrich et al. structured their analysis following four progressive levels of population comparisons:

• Industrialized societies versus small scale societies. For various reasons, including environmental adaptation, members of industrialized societies frequently score as outliers when comparing some of their (basic) psychological functions to those of members of various small-scale societies around the world. Consistent variations in visual perception (pp. 64–65), economic decision-making (pp. 65–67), e.g., social motivation, fairness, folk-biological reasoning (pp. 67–68), and spatial cognition (pp. 68) are reported, and others in decision-making are also likely to exist.

• WEIRD versus non-WEIRD societies. Consistent variations are reported with regards to social-decision making (e.g., fairness, cooperation, punishment), reasoning strategies (with a tendency of Westerners to be more analytic, and of others to be more holistic), moral reasoning, and to independent/interdependent self-concepts. Here, Westerners have a tendency to be more individualistic than others (which has implications for several psychological features such as attention, motivation, situational appraisal and emotions, positive self-view, personal choices, and motivation to conform).

• Contemporary Americans versus the rest of WEIRD societies. As expressed in Arnett’s work, American research content constitutes a major share of contemporary psychology production, but additional research indicates that US people have characteristics that make them different from other WEIRD populations, such as a higher tendency for expressing strong individualism. Henrich et al. suggest that this situation may be the illustration of an ideology that “particularly stresses the importance of freedom and self-sufficiency, as well as various practices in education and childrearing” (p. 74) which enforces individualism. According to Henrich et al., the American emphasis on individualism and independence influences their choice-making approaches, and a tendency to be exceptionally analytic as well. Henrich et al. also mentioned sociologist Seymour Martin Lipset (1996), who documented that among other things, Americans were “the most patriotic, litigious, philanthropic and populist,” and among “the most optimistic and least class-conscious.”

• Typical contemporary American subjects versus other Americans. As stated in Arnett (2008), a significant share of American psychology findings have been obtained from sampled populations of psychology undergraduates. Henrich et al. reported studies showing that, when compared to other Americans, college students tend to have more rational choices, higher scores on individualism, weaker conformity motivation, and less negative views about group and racial diversity within the society. They are likely to be embedded in less tightly structured social networks, to be less interdependent and less holistic, and relate more to an ethic of autonomy for moral reasoning. Research in behavioral economics also suggests that they are consistently less pro-social than other Americans, as illustrated by results considering several psychological features (e.g., trust, fairness, cooperation, punishment of unfairness, free riding). When sample subjects are children, studies reported by Henrich et al. suggest that participants are likely to have parents with a high Socio-Economic Status (SES). This can be linked to other studies showing that low-SES and high-SES children show differences in psychology-related processes such as spatial reasoning.

The conclusion that Henrich et al. stressed of their survey is that WEIRD subjects are frequent outliers in the World population, that they “may often be the worst population from which to make generalizations” (p. 79), and that the demonstrated extreme reliance on WEIRD samples “may cause researchers to miss important dimensions of variation, and devote undue attention to behavioral tendencies that are unusual in a global context” (p. 80). While acknowledging that exclusive reliance on WEIRD samples may be adequate when seeking existential proofs (i.e., that a process exists), they strongly call for more comparative studies and insist that the consideration of human diversity has to become more systematic in psychology.

Additional Commentaries on Henrich et al. (2010a)

The review of Henrich et al. has been commented on by many researchers, a large majority of them supporting its conclusions (Various authors 2010). Scholars from different fields also proposed additional elements and issues to consider. The following paragraph summarizes some of the initial comments that the paper received.⁴

For similar reasons as those highlighted by Henrich et al., Lancy (pp. 99–100) extended the suspicion of WEIRD biases to cognitive development, children’s social behavior, and parent–child interaction. Chiao and Cheon (pp. 88–90) report evidence of socio-cultural variations in brain functioning. Stich (p. 110) suggests that philosophical production and intuitions are subject to WEIRD-biases. Similarly and as mentioned earlier, Baumard and Sperber (pp. 84–85) note that the high dominance of WEIRD scholars results in WEIRD-flavored experimental designs that could be interpreted in a different way by non-WEIRD populations, thus altering results. Majid and Levinson (p. 103) insist that English and other WEIRD languages introduce distortions in research. Ceci, Kahan, and Braman (pp. 87–88), as well as Kesebir, Oishi, and Spellman (pp. 96–97), advocate for considering more contextual and socio-ecological information in order to better detect principles that could be generalized. Fernald (pp. 91–92) suggests focusing on social variations for correctly addressing the domain of early development of fundamental cognitive and language capacities. Fessler (p. 92) thinks that investigators would be more efficient at detecting culturally variable features if they were to study samples that are culturally different from themselves. Finally, Rochat (pp. 107–108) lists necessary aspects to consider for adequate population sampling, and for ensuring the validity of measuring instruments, while Gosling, Carson, John, and Potter (pp. 94–95) insist on “the promise of the Internet in reaching more diverse samples” (p. 94).

Investigating the Socio-Cultural Status of AIED Research

As a field interested in overcoming educational challenges with adaptive technologies, AIED has been strongly influenced by psychological research and findings. Many psychology theories have inspired and continue to influence the design and development of AIED systems. Consequently, the two following research questions are investigated in this section:

• Do detected socio-cultural imbalances in psychology have implications for the AIED research field?

• Is AIED itself inclined to produce WEIRD-imbalanced research?

Do Detected Socio-Cultural Imbalances in Psychology Have Implications for the AIED Research Field?

Many AIED topics are strongly influenced by research in psychology, but it could be claimed that they are all about basic and universal human processes. To reject this hypothesis and confirm that cultural variations exist for psychology-related topics of interest to the AIED community, we first concentrate our investigation on ‘emotions,’ since this is currently one of the most popular research topics in AIED. There is indeed evidence that affective experiences vary across cultures in many ways.

Though universalisms in emotion have been identified (Ekman 1972),⁵ cultural variations in various emotional features have also been extensively reported (e.g., Matsumoto and Ekman 1989). According to a literature review (Mesquita et al. 1997), numerous studies have identified cultural variations in affective antecedents (events or objects that trigger an emotion), subjective experiences (feelings), behavioral response, in physiological changes related to an affective experience, and in affective appraisal. Considering more specifically the last point, members of a cultural group commonly develop and internalize specific interpretations about a situation (Sharifian 2003). Hence the experience of a similar situation by culturally distinct peoples can result in different interpretations, and therefore in different affective reactions. For example, stereotypes are socio-cultural constructs that ‘simplify’ cognitive and behavioural reactions to situations (Jost and Hamilton 2005), and many of them have strong affective implications at different levels. Furthermore, the frequency in reporting positive or negative emotions is also known to vary across cultures, and cultural differences in processes pertaining to emotion recall have also been reported (Robinson and Clore 2002; Scollon et al. 2004). This implies that post-hoc assessments of affect may thus be culturally-marked. As a conclusion to their survey, Mesquita et al. claim that it is “convincingly demonstrated that there are cultural differences in the ecology of emotions” (p. 265).

Furthermore, research has identified cultural variations in educational practices and teachers’ beliefs (TALIS 2009), as well as in many other psychological topics of interest for the AIED research community, such as decision making, reasoning, motivation, self-regulation, collaboration, competition, personal values, and various aspects of perception and interpretation (Henrich et al. 2010a; Various authors 2010; Purdie and Hattie 1996; Lynch et al. 2009; Ryan et al. 2005; Hofstede 2008).

No evidence suggests that research on emotion and on other psychological topics of interest for AIED is informed by more culturally-aware practices. Hence it can be fairly assumed that psychological research that influences AIED is similarly WEIRD-imbalanced and that its cultural variability is mainly ignored.

Through transitivity, AIED research and activities are thus likely to be biased by this situation, which can manifest in many ways such as:

• A lack of consideration of culture as a relevant factor to explain user responses to AIED systems. This is especially important when collecting data to inform human features that are known to be culturally sensitive such as self-concepts, motivation, or emotion.

• A lack of interest in or an ignorance of educational issues in non-WEIRD societies that could benefit from AIED paradigms.

• The dominance of system and interaction designs that may be appropriate for WEIRD audiences but less optimal for other populations. This could actually result in additional challenges for non-WEIRD scholars wishing to integrate with the AIED community. It may be more difficult for these scholars to obtain similarly effective systems when grounding their research and developments on paradigms that are possibly less natural and appropriate for their local environment. This can therefore result in more negative results and less successful systems, which are harder to publish.

Overall, problems resulting from cultural imbalances in psychology are likely to similarly apply to the AIED research field (see the previous section on implications of the detected USA and WEIRD imbalances). However, there might be a balancing effect to this bias. AIED is a technological research field and it penetrates societies as information technologies do. Given the existence of a Digital Divide phenomenon, AIED systems that have gone beyond laboratory testing in the last few decades have most often been deployed in WEIRD environments and especially in US institutions and organizations. Consequently, biasing effects induced by socio-cultural imbalances in psychology might have been somewhat limited since typical AIED end-users would be mostly similar to typical subjects of psychology research so far.

Nevertheless, and as reported by Henrich et al. (2010a), socio-cultural variations exist and are identified between (and within) the different WEIRD societies as well as between typical psychology subjects and the rest of the WEIRD population. Despite AIED end-users being mainly from WEIRD countries and especially from the USA, psychological findings obtained from specific populations may still have influenced AIED systems targeting other populations even though the appropriateness of such a generalization may not have been established at all.

As a summary, it can be fairly assumed that socio-cultural imbalances detected in psychology have a biasing effect on past and present AIED research activities, but these negative influences might have been limited by the fact that typical AIED end-users could have been most of the time US and more generally WEIRD individuals so far.

Is AIED Inclined to Produce WEIRD-Imbalanced Research?

The fact that issues in psychology spread to and influence AIED activities is relatively obvious considering that psychology is historically important for this research field. However, the AIED community is not only a consumer of external research findings, it also produces its own studies, norms, practices, and theories. This section thus investigates whether AIED is affected by similar socio-cultural imbalances to those detected in psychology, and is likely to produce culturally-biased research.

Methodology

The methodology employed here was largely inspired by the one described in Arnett (2008), and consisted of analyzing AIED research. Consequently, full paper publications in two AIED-related international conferences were analyzed over a period of 12 years (2002–2013). Selected conferences were the International Conference on Intelligent Tutoring Systems (ITS) and the International Conference on Artificial Intelligence in Education.

A conference-focused analysis was preferred to the journal-based approach of Arnett because the AIED community includes scholars from several different disciplines. Therefore journal publications dedicated to AIED research are spread not only among the few journals dedicated to the field but also among many others that are discipline-specific.

AIED and ITS conferences were chosen because their content is expected to correctly reflect state-of-the-art AIED research production.⁶ They are commonly acknowledged as the two top-tier conferences for AIED research, and are perceived as major yearly events for the AIED community. Although other conferences of quality exist (e.g., ICCE, ECTEL, ICALT), it is commonly agreed that no other event has achieved a similar level of international recognition in the AIED field.⁷

Both selected conferences are biennial: ITS2002 occurred in France and Spain, AIED2003 in the USA, ITS2004 in Brazil, AIED2005 in the Netherlands, ITS2006 in Taiwan, AIED2007 in the USA, ITS2008 in Canada, AIED2009 in UK, ITS2010 in the USA, AIED2011 in New Zealand, ITS2012⁸ in Greece, and AIED2013 in the USA.

Origins were coded according to the following categories: USA and English countries (Canada, Australia, New Zealand, UK, and Ireland) were considered separately as in Arnett (2008) whereas other categories were related to continents: Europe, Asia,⁹ and Africa.¹⁰ A Latin America category was also considered and was essentially composed of entries from South America and Mexico.¹¹

National affiliations of first authors, of other authors, and national origins of samples were recorded, and the rest of the coding approach was similar to Arnett (2008). If an author had several affiliations in the same category or if a paper included several samples from the same origin, only one entry was recorded. However, if an author had institutional affiliations in multiple categories or if a paper included several samples related to different categories, one entry was recorded for each related categories. This occurred on extremely rare occasions.

The following sections present records of international representations in AIED authorship and AIED samples.

International Representations in AIED Authorship

The following tables present national affiliations of first authors (Table 4) and other authors (Table 5) in the twelve last ITS and AIED conferences.¹²

Table 4 National affiliation of first authors in ITS and AIED conferences

Table 5 National affiliation of other authors in ITS and AIED conferences

Results reported in Tables 4 and 5 illustrate very strong WEIRD-imbalances in authorship, although most of the time these are slightly lower than those identified in psychology (Tables 1 and 2). First authors and other authors are affiliated to US and WEIRD institutions to a very large extent. Latin American and Asian authors have extremely low representations and, furthermore, almost all of them come from very limited subsets of countries: essentially Brazil and Mexico for Latin America, and Japan, South Korea, Taiwan, and Singapore for Asia. Continental China and India have almost no author representation even although they together account for one third of the World population and have a growing importance in the global economy. No first authors from African institutions are reported and very few other authors that all come from the Maghreb or Egypt.

International Representations in AIED Samples

Because of the interdisciplinary nature of the field, some papers published in ITS and AIED conferences do not include human-based evaluations whereas other papers use human-based samples and datasets exclusively to validate technical features such as machine-learning or clustering algorithms. Only papers with evaluations that did (at least partially) reflect upon human features were considered. Datasets could be considered as long as data collection was appropriate for socio-cultural investigations, even if they had not been collected for a specific study and rather came from a data repository. Evaluations based on simulated students were discarded because they model only a limited set of characteristics of human beings, and there is no demonstration they exactly mimic complex aspects of human students such as cultural features.

Figure 1 presents the percentage of papers with either human-based evaluations or investigations of human features in the twelve considered conferences. It clearly shows that the frequency of papers respectively with human-based evaluations and investigations of human features have both notably increased in recent conferences.

Fig. 1

Percentage of papers including human-based evaluations and investigations of human features in ITS and AIED conferences

The goal of the next part of the analysis was to report origins of AIED samples of interest, i.e., those used to investigate human features. However a significant share of these samples did not include clear information about the country of origin but, in most cases, it was possible to safely infer their origins by cross-checking indirect clues. Nevertheless, a few additional samples were discarded because of the impossibility of determining their origin with sufficient confidence. Table 6 presents the national origin of the remaining samples of interest from the 12 ITS and AIED conferences.

Table 6 National origin of relevant samples in ITS and AIED conferences

Results in Table 6 indicate US and WEIRD imbalances similar to those reported in psychology (Table 3) although the “English countries” cluster accounts for a greater share of samples. Asia is slightly more represented in relevant AIED samples than in psychology ones. There are almost no relevant South American samples, and absolutely none from Africa.

Discussion and Analysis by a Panel of AIED Experts

Methodology and Panel Presentation

In order to assess these results in a non-dogmatic way, several scholars with strong AIED expertise¹³ were contacted. A first group of 7 experts whose comments were mentioned in an earlier-yet-limited version of this work (Blanchard 2012a) was later completed by 5 additional experts.¹⁴ They were all presented with the same information in the exact same way. It consisted of a word document with a shorter overview of Arnett (2008) and Henrich et al. (2010a), followed by data collected from ITS2002 to AIED2011, inclusively, since data for ITS2012 and AIED2013 became available after the release of this first paper version. The document was followed by an open-ended questionnaire to assess a) how panel members analyzed the data, and b) whether they agreed to some suggestions for addressing the situation.¹⁵

In this panel of 12 experts, there is an equal number of men and women. Regarding their academic affiliation, 3 are from the USA, 4 are from English countries, 3 are from Europe, and 2 are from Asia. Regarding their academic background, 3 report a PhD in AIED/educational technologies, 2 in psychology, 3 in computer science, 2 in human-computer interaction, and 1 in artificial intelligence. Since the objective was to obtain feedback from the current AIED community, it was important to have a representative panel. Origins of affiliations seem to fairly represent typical attendance of ITS and AIED conferences even if, despite numerous attempts, no targeted South American experts positively replied to the invitation. No expert with an African affiliation and meeting the AIED expertise criteria could be identified. Finally, the composition of the general community evolves over time and there is no definitive information about discipline representation within the conferences. However, efforts have been made to select experts to represent both the technical, social, and humanities dimensions of the field.

The following section summarizes views and comments of panel members that were collected individually. Specific care has been put into fairly illustrating as many of the expressed opinions as possible, which means that some of these positions are not necessarily shared by other panel members or the author.

Panel Commentaries

All experts acknowledged that the collected data demonstrate strong USA and WEIRD dominance in AIED research. For most of the experts (and for the author as well), it was important to insist that the detected imbalances are unintentional, and that the selection of papers is only based on scientific criteria.

All experts agreed to different extents that these imbalances are very likely to influence AIED activities, although some of them rightfully insisted that no direct results are actually provided on how these imbalances may be impacting AIED, and consider that it may be too early to use the word ‘bias.’ Several panel members considered the AIED field to have several important differences when compared to psychology, and believed that the imbalances would have implications that are unique to this community. They suggested therefore that investigations to identify these effects should be undertaken: more master and doctoral students should start projects to investigate how culture and AIED intersect, and more AIED-related events should include cultural considerations in the future.

Several experts from different origins reported the same following position: a US bias in AIED is not surprising because an American-centric view of the World is affecting many other areas of daily life in many other societies and, according to one expert, there is a worrying “strong tendency to blindness to that bias.” Although the influence of the US culture on other societies is likely to be the strongest, several other countries are also culturally influencing other societies (e.g., France vs. several French-speaking countries).

Another expert noticed that AIED research is already reaching various socio-cultural groups, within the USA especially, according to ethnic, economic, and localization criteria, and consequently claimed that “AIED is doing better at involving diverse populations than [the reported data] implies.” Several other experts further insisted that beside socio-cultural representations, other potential sampling imbalances in AIED should be investigated as well.

Panel members were also asked to reflect on several actions to potentially mitigate the impact of detected socio-cultural imbalances. Most experts agreed that in order to achieve more culturally-aware AIED research, the AIED community needs to be made aware of the situation as a necessary first step. This was indeed the main objective of this paper. Positive improvements should first and foremost come from community members better at self-regulating their research efforts, e.g., by including socio-cultural considerations more frequently in their research design and analyses. Likewise, experts agreed that there is place for improving sample descriptions. For example, authors should more systematically describe appropriate contextual factors such as nationality, socioeconomic status, ethnicity, gender, etc.¹⁶ whereas reviewers could more frequently highlight ill-presented samples. The considered population should also be more commonly highlighted in paper conclusions.

Several experts agreed that organizing conferences more frequently in non-WEIRD societies would help raise the diversity of the community. Indeed, it is to be noticed that of the 12 considered conferences, only 2 occurred in non-WEIRD societies: ITS2004 in Brazil and ITS2006 in Taiwan, and these conferences respectively had the highest rate of papers from Latin America and Asia (see Table 4). More generally, conference localization almost always impacted positively on the representation of researchers from surrounding countries. However, another expert insisted that conferences should mainly be organized in countries that are easy to reach by researchers from established AIED countries. Otherwise the overall quality, attendance,¹⁷ and consequently the long term credibility of conferences could be threatened. This position is not necessarily in opposition with a more frequent localization of conferences in non-WEIRD countries. There are many intertwined factors that influence the decision of organizing a conference in a specific location. A decisive one is that WEIRD scholars may be more experienced at submitting conference organization proposals which are consequently more successful.

Many panel members suggested that the AIED community should question itself about research it should focus on developing. Several experts reported that investigating socio-cultural variations (or universalisms) in different contexts is not perceived as bringing ‘new’ findings by some reviewers. They invite the community to reflect upon this point. Another expert posited that reported cultural imbalances would not be an issue if the AIED community correctly followed the ‘scientific paradigm,’ which (s)he claimed is not currently the case.

Finally, several panel members with different academic profiles also criticized the current importance of human-based evaluations on paper acceptance/rejection decisions although they do not question their general interest for the field. Some of them consider that an evaluation, even if loosely done, now has too much impact on paper acceptance. They see this situation as very problematic since a paper only detailing a clever technical solution might sometimes be a better contribution to the field although it seems harder to get published. Continuing into this direction, one of the experts with a background in humanities argued that over the past decade, AIED has been distracted from its innate quest of investigating “how to automate interaction in a teaching/learning environment,” or how to “develop representations and algorithms by means of which [educational] software can adapt their interaction.” Instead there are more and more studies attempting to investigate “truly psychological issues related to education” that (s)he claims should remain the prerogative of other disciplines “such as Developmental Psychology, etc.” Other panel members have produced similarly virulent critiques based on a perception of a growing importance for human evaluations in the paper selection process.¹⁸ A panel member also reported the case of an AIED senior member focusing on this point when commenting on Blanchard (2012a) during an academic event.

General Discussion

This section provides additional culture-related insights that result from several years spent investigating cultural disciplines and theories, and from interactions with various scholars interested in the emerging interdisciplinary field of culturally-aware computing.

Non-WEIRD Societies as Emerging AIED Markets with High Potential

As it has been stated, negative effects of identified socio-cultural imbalances in samples might have been limited by the fact that typical past and current AIED end-users might mainly be WEIRD and US individuals, the very cultural clusters that have been mostly investigated. However, the Digital Divide phenomenon is rapidly fading in the twenty first century. IT technologies are becoming more ubiquitous in both developed and developing non-WEIRD countries, and this could bolster the global dissemination of AIED systems.

Nevertheless, the needs of WEIRD societies for AIED technologies remain and so does the need to keep addressing them. The main objective in these contexts is to improve established educational systems and opportunities. However, in non-WEIRD societies, challenges that AIED will have to tackle could be slightly shifted. Of course, AIED should contribute to the improvement of existing educational systems of non-WEIRD countries as well. But many of these societies have been and keep experiencing massive societal changes. With the rapid growth of middle-classes in demographic giants such as China, India, Brazil, the Philippines, or even Nigeria to name but a few, the most important challenge for AIED may thus rather consist of offering educational opportunities that are currently missing, and addressing the lack of efficient educators in many disciplines and at different levels.

These are great opportunities for the AIED community to have a positive life-changing impact on many human beings, and for AIED systems to reach new economically-viable markets. The way the AIED community decides to tackle this challenge will have its importance. It may choose to consider that these new audiences do not differ from those already researched, and thereby adopt an approach that is close to neo-imperialism, or it may develop and follow guidelines and practices to better consider and respect cultural diversity and specificities (see for example UNESCO 2007).

Developing AIED Investigations in Foreign Cultures

More studies should investigate AIED challenges on samples representing different populations than those of the USA and other WEIRD countries. However, there is a lack of experienced AIED researchers in many non-WEIRD countries and consequently investigations on the related populations will largely depend on initiatives of WEIRD scholars in the short term. Setting up an evaluation on a sample from a distinct culture is not a trivial task and may provide data that seems to initially answer a target question although they actually inform other features.

For example, testing a system or a methodology on other culturally-distinct populations does not necessarily provide information about its universality. A population from a country Y may be commonly acquainted with aspects of the culture of a country X. When testing if a system S developed to suit X is correctly used by population Y or if it produces similar results, one rather investigates features related to the cultural intelligence (Earley and Mosakowski 2004) of individuals from the Y population, that is, whether their knowledge of the X culture allows them to understand and interact with S. A successful usage of S by population Y, or the report of positive emotions in this situation, are both interesting results and could inform several research questions, but without further investigation they neither demonstrate that S is similarly adequate or optimal in countries X and Y, nor that some of the features of S do not seem odd to population Y. Analyses that would investigate whether WEIRD-produced tools and questionnaires can be used or answered by a distinct population rather than the original one they were conceived for would also be more likely to assess cultural intelligence features.

As it has been stated earlier in this paper, different cultures lead groups to endorse different dominant ecologies of conceptualization (see Sharifian 2003). Studies on culturally-specific conceptual frameworks related to education at large have been relatively limited so far. Hence, exploratory investigations that are open-ended and that do not focus on assessing WEIRD-defined constructs are greatly needed and should open many new and exciting research trends for the AIED community. As suggested by Gosling et al. (Various authors 2010, pp. 94–95), increasing global access to the Internet could facilitate such data collection, but the AIED community could also propose and develop dedicated technologies to reach diverse human populations, and procedures for controlling data collection.

A particular challenge is to design and produce systems dedicated for a target population. It may be essential for researchers to first develop a thorough expertise of the target population such as through reading ethnographic, anthropological or sociological surveys. However, and as it has been stated before, the cultural variance of features is frequently hidden and subconscious, and it may be impossible to achieve a high level of confidence about the absence of cultural oddness unless proper cultural representatives are involved.

Mentoring Programs to Increase Diversity

As discussed previously, investigations led by US and WEIRD AIED scholars naturally insist more on research topics of interest in US and WEIRD contexts (see Various authors 2010, pp. 84–85). However, many non-WEIRD societies are emerging markets for the AIED research field, as explained previously. These cultural contexts present education-related challenges that are totally absent from US and WEIRD societies, although these issues may be critical for raising the interest of non-WEIRD populations toward AIED technologies and facilitating their adoption in new contexts. There is no better way to identify these hidden needs than to have more AIED members from underrepresented societies. More diversity within the AIED community could also result in the emergence of alternative solutions to those typically proposed by WEIRD researchers.

However, potential community members may not be accustomed to the specific culture of the AIED community or to various challenges pertaining to the peculiar task of writing solid scientific papers. Making the same observation for psychology, Arnett suggested reviving a program where experienced authors develop a mentorship relation with community outsiders and/or newcomers. These mentors can suggest steps to improve the quality of research production to specific community standards. Developing a similar program in conjunction to AIED-related events has been similarly suggested by various members of the community.

Evolution of the AIED Culture

The description of an increasing share of conference papers using human evaluations shown in Fig. 1, as well as the related comments in the last paragraph of the panel section, both show that the culture of the AIED community has evolved in the last decade. Given the current rate of human-based evaluations in accepted full papers (generally close to or over 90 %), the inclusion of such a feature in paper submissions can be considered a de facto norm that emerged over time. This is a typical example of a cultural evolution process (see Henrich and McElreath 2007, for more details). This evolution is not consensual, however, and comments of some panel members illustrate that frustration exists toward it within the AIED community.¹⁹ Detractors of this evolution especially argue that it provides unfair advantages to people with expertise in humanities who are trained for developing and presenting human-based evaluations. They also suggest that many technological research areas have weaker chances of appearing in proceedings given the importance that some reviewers give to human-based evaluations. Promoters on the other side insist on the human dimension of AIED and reason that demonstrating the effectiveness of learner-centered technological solutions without these evaluations is impossible.

This paper cannot settle the argument on whether this evolution is for the greater good of the domain. Nevertheless, from a cultural standpoint, this state of affairs presents risks quite similar to others that have been discussed in this paper. Researchers with expertise in humanities, being overall more skilled at preparing and presenting human-based results, may encourage greater representation of humanities research topics to the detriment of other topics that are conveyed by scholars with a more technological profile. This is particularly relevant for scholars from countries where technological curricula do not commonly include training in disciplines such as social science statistics, or where academic systems and institutions make interdisciplinary teams harder to establish.

Options for addressing this situation may also be relatively similar to those related to international representations in psychology or AIED. The community could consider that this norm is adequate, that detractors are a minority, and just ignore their position. It could also still consider that the norm is adequate but acknowledge that it disadvantages a creative part of the community and the specific research questions it conveys. To mitigate this effect, mentoring strategies could be developed to support researchers less skilled at developing and presenting human-based evaluations. Another option for the community would be to consider the norm as inadequate and propose measures to control the importance of human-based evaluations. For example, conference tracks targeting humanities-related papers, technology-related papers, or hybrid papers could be proposed and implement different paper selection approaches.

Conclusion

During the writing of this paper, extreme care was put into providing an accurate presentation of data on international representations in AIED authorship and sampling. The section summarizing the commentaries of a panel of AIED experts was written with a dedicated attentiveness towards achieving a balanced and honest description of all the contributions.

Culture is a sensitive topic, and researchers all convey a personal ecology of interpretations. Despite considerable effort to limit this effect, which includes the collection of commentaries by researchers with different cultural backgrounds, it is acknowledged that personal positions of the author may have inadvertently influenced the design and writing of this paper, just as it may influence the interpretations drawn by some readers. Therefore, final clarifications are provided in this section to limit misunderstandings.

It cannot be ignored that the data described in this paper could be used to nurture a dangerous struggle for influence between schools of thoughts that could only be detrimental to AIED research. However, the author strongly believes that the community holds all the wisdom it needs to ensure that the suggested debate remains productive and focused on the improvement of the field. Data provided in this paper could indeed be used to ground fair and informed discussions:

• first and foremost on the limited cultural diversity of the AIED research field and whether actions have to be undertaken for its greater good,

• but also on the current culture of its interdisciplinary community and whether its evolution is appropriate for efficiently tackling all the possible research questions related to it.

Overall the position advocated in this paper is threefold:

• detected imbalances are dangerously strong and may pose a threat to the quality of the AIED research.

• ‘quality’ must remain the main determinant of research acceptance and diffusion, but its definition changes between disciplines, and it may also have culturally-variable interpretations as well,

• a consensual situation for AIED to better consider diversity in its various facets can be reached with more involvement of the community.

It must also be clarified that it has never been the objective of this paper (nor the one of Arnett 2008 or Henrich et al. 2010a) to suggest that research production should be constrained, such as for example by country quotas. One should indeed keep in mind that observed imbalances in AIED are the result of unconscious processes, economical differences, and historical aspects among other factors. This partly explains that imbalances in research production are in the natural order of things. However, they should be better controlled and moderated.

It is finally important to insist that the WEIRD vs non-WEIRD dichotomy, the main granularity level used in this paper (and in Arnett 2008 and Henrich et al. 2010a) for discussing cultural differences, has mainly been selected for reasons of convenience. It is a satisfying-enough approach for supporting the message of this paper, that culture should be better taken into account in AIED. However there are no claims that it is the best perspective to embed cultural awareness in AIED systems, nor that discussing cultural differences from a country or country-cluster standpoints are the sole valid approaches since cultural groups vary in criteria that define them (see Blanchard 2012b).

Overall this paper probably brings many more questions than answers, and this is a deliberate approach. There is currently a lack of studies and results on the real impact and influences of cultures on the AIED field. Providing assumptions and claims at this stage would thus be likely to end up in the expression of cultural stereotypes, which would be detrimental to this emerging research trend that is obviously in an exploratory phase.

Indeed, appropriately addressing cultural diversity is possibly one of the most complicated AIED topics to consider (see Blanchard and Mizoguchi 2014; Blanchard and Ogan 2010). It will require time and systematic analyses. It also requires that researchers avoid jumping to conclusions too easily. But the possibility to have a universal impact on human beings is real, and should be appealing to members of the community.

Finally, and to paraphrase a former US president, the AIED community should choose to embrace this challenge not because it is easy, but because it is hard, and because it could serve to organize and measure the best of the energies and skills of this community.